amd64/108861: nve(4) driver on FreeBSD 6.2 AMD64 does not work at 1Gbps with nForce4 NIC

[Lawrence Stewart]

>Number:         108861
>Category:       amd64
>Synopsis:       nve(4) driver on FreeBSD 6.2 AMD64 does not work at 1Gbps with nForce4 NIC
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-amd64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Feb 07 03:20:18 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator:     Lawrence Stewart
>Release:        6.2
>Organization:
Swinburne University
>Environment:
raidserv2# uname -a
FreeBSD raidserv2.caia.swin.edu.au 6.2-RELEASE FreeBSD 6.2-RELEASE #1: Wed Feb  7 11:30:15 EST 2007     root at raidserv2.caia.swin.edu.au:/usr/src/sys/amd64/compile/GENERIC+SMP  amd64

>Description:
The research centre I work at recently purchased a new file server, built from commodity hardware that FreeBSD claimed to have support for.

Motherboard is an Asus A8N-E Rev 2.00 (listed as supported here: http://www.freebsd.org/platforms/amd64/motherboards.html).

I clean installed FreeBSD 6.2 from ISO 6.2-RELEASE-amd64-disc1.iso.

The onboard nVidia nForce4 nve(4) based NIC was plugged into the university's Cisco based gigabit ethernet LAN via CAT5e straight-through patch lead.

An Intel 82540EM em(4) based PCI NIC was also installed in the server, but was not plugged into a network point (we previously tried installing FreeBSD 6.1 which had no support for the onboard NIC, so we put the Intel card in to use. We left it in for the new test with FreeBSD 6.2).

The install completed successfully. We didn't configure either NIC during install at all (well... that's not entriely true the first time through... but I'll save the details of that crazy story for later as it will confuse things for the moment).

After reboot, we bought the NIC up using "ifconfig nve0 up" and it synced at 1000baseTX full-duplex. Tcpdumping on the interface shows no activity at all (even though the uni network is VERY chatty, multiple packets per second should have been showing). Running "dhclient nve0" shows outbound DHCP requests, but not any return packets. 

Forcibly syncing the interface to 1000baseTX using "ifconfig nve0 media 1000baseTX mediaopt full-duplex" resulted in same behaviour i.e. doesn't "see" inbound packets. Forcibly syncing the interface to 100baseTX all of a sudden shows packets being received in tcpdump, and we can use the interface, though it only achieves about 80kb/sec on transfers i.e. is severely restricted for some reason.

Intrigued, I installed a small 5 port 100baseTX switch between the server and network port and set the interface back to autoselect using "ifconfig nve0 media autoselect". The nve(4) interface synced at 100baseTX, and could then transfer at full 100Mbps speed (scp @ 11MB/sec from another local computer).

I tried resolving the issue by setting "hw.pci.enable_msi=0" and "hw.pci.enable_msix=0" in /boot/loader.conf as suggested for fixing em(4) related problems. It had no effect from what I could see. I also tried rebuilding and installing a GENERIC kernel, GENERIC kernel without nve driver and kldloading it after startup, GENERIC+SMP kernel... none of the different kernel configs seemd to change anything.

I also might add that I tried getting the em(4) Intel NIC working when I realised the nve(4) NIC was going to be a pain, and I had no luck, even with the disabling of MSI in /boot/loader.conf. I didn't try any harder than that though because my main focus was seeing if the nve driver was fixed.
>How-To-Repeat:
Patch the onboard ethernet NIC on an Asus A8N-E Rev 2.00 motherboard into a 1GBps switch port.

Install FreeBSD 6.2 from ISO 6.2-RELEASE-amd64-disc1.iso.

Configure the NIC to autoselect it's sync speed using "ifconfig nve0 media autoselect".

Bring the NIC up using "ifconfig nve0 up"

It should sync at 1000baseTX full-duplex, but not be able to communicate at all via the interface.

>Fix:
I could not find a fix for the problem.

I also could not figure out why explicitly setting the NIC to sync at 100baseTX full-duplex made the NIC work, but only at a fraction of 100Mbps capacity.

A work around is to cause the switch port the NIC is plugged into to only allow 100baseTX, so that the NIC will autoselect at 100baseTX and work at reasonable speed. You could also do as I did and stick small 100baseTX switch in between the machine and gigabit switch.
>Release-Note:
>Audit-Trail:
>Unformatted: