ste(4) NIC's RX ring head may get ahead of the driver [PATCH]

[Ruslan Ermilov]

Hey Doug,

I'm writing to you because you were the last who touched this driver
seriously, but since it's been 1,5 years ago, I'm also Cc:ing the
freebsd-net mailing list, as I'm not sure if you're still interested
in this driver.

To make the long story short, under a heavy RX load, the ste(4) NIC's
RX ring head may get ahead of what driver thinks, bringing all sort
of havoc like stuck traffic, disordered packets, etc.  The NIC never
gets out of this state, and the only workaround is to reset the chip,
and so we did for some time (by adding the IFF_LINK2 handler to call
the driver's watchdog function).

A similar problem is known to exist with other NICs, such as dc(4)
and xl(4), and their drivers have workarounds for this situation.

We've adopted the approach used by dc(4) and xl(4), but instead of
seeing if we need to re-synchronize the head _after_ receiving (like
dc(4) and xl(4) drivers do), we do it at the beginning of ste_rxeof().
As statistics shows, the number of resyncs needed is smaller by a
factor of 3 or more in this case, because often the RxDMAComplete
interrupt is generated when RX ring is completely empty(!), and as
NIC continues to do DMA and fill the RX ring while we're still
servicing the RxDMAComplete interrupt, we did more resyncs than was
actually necessary.

Also, we were able to further reduce the number of resyncs by setting
the RxDMAPollPeriod to a higher value.  320ns looked like an overkill
here, and I'm not sure why you have chosen it in the first place,
when adding polling support for RX in the driver.  Also, we believe
that this setting may be responsible for what you referred to as:

> This card still has seemingly unfixable issues under heavy RX load in
> which the card takes over the PCI bus.

in the commit log for revision 1.33 of if_ste.c.

Attached is the patch (for RELENG_4) we're currently using, and are
quite happy with.  If anyone is using ste(4) NICs and is experiencing
similar problems, I'd be glad to hear the reports about this patch.


Cheers,
-- 
Ruslan Ermilov
ru at FreeBSD.org
FreeBSD committer
-------------- next part --------------
Index: if_ste.c
===================================================================
RCS file: /home/cvs/ipnet/freebsd/src/sys/pci/if_ste.c,v
retrieving revision 1.1.1.2
retrieving revision 1.7
diff -u -p -r1.1.1.2 -r1.7
--- if_ste.c	29 Mar 2004 11:47:42 -0000	1.1.1.2
+++ if_ste.c	29 Mar 2004 13:41:47 -0000	1.7
@@ -30,6 +30,7 @@
  * THE POSSIBILITY OF SUCH DAMAGE.
  *
  * $FreeBSD: src/sys/pci/if_ste.c,v 1.14.2.10 2004/03/25 08:49:22 ru Exp $
+ * $IPNet: ipnet/freebsd/src/sys/pci/if_ste.c,v 1.7 2004/03/29 13:41:47 ru Exp $
  */
 
 #include <sys/param.h>
@@ -39,6 +40,7 @@
 #include <sys/malloc.h>
 #include <sys/kernel.h>
 #include <sys/socket.h>
+#include <sys/sysctl.h>
 
 #include <net/if.h>
 #include <net/if_arp.h>
@@ -163,6 +165,10 @@ static driver_t ste_driver = {
 
 static devclass_t ste_devclass;
 
+static int ste_rxresync;
+SYSCTL_INT(_hw, OID_AUTO, ste_rxresync, CTLFLAG_RW,
+	&ste_rxresync, 0, "");
+
 DRIVER_MODULE(if_ste, pci, ste_driver, ste_devclass, 0, 0);
 DRIVER_MODULE(miibus, ste, miibus_driver, miibus_devclass, 0, 0);
 
@@ -691,6 +697,19 @@ static void ste_rxeof(sc)
 
 	ifp = &sc->arpcom.ac_if;
 
+	if (sc->ste_cdata.ste_rx_head->ste_ptr->ste_status == 0) {
+		cur_rx = sc->ste_cdata.ste_rx_head;
+		do {
+			cur_rx = cur_rx->ste_next;
+			/* If the ring is empty, just return. */
+			if (cur_rx == sc->ste_cdata.ste_rx_head)
+				return;
+		} while (cur_rx->ste_ptr->ste_status == 0);
+		/* We've fallen behind the chip: catch it. */
+		sc->ste_cdata.ste_rx_head = cur_rx;
+		++ste_rxresync;
+	};
+
 	while((rxstat = sc->ste_cdata.ste_rx_head->ste_ptr->ste_status)
 	      & STE_RXSTAT_DMADONE) {
 		if ((STE_RX_LIST_CNT - count) < 3) {
@@ -1255,7 +1274,7 @@ static void ste_init(xsc)
 	}
 
 	/* Set RX polling interval */
-	CSR_WRITE_1(sc, STE_RX_DMAPOLL_PERIOD, 1);
+	CSR_WRITE_1(sc, STE_RX_DMAPOLL_PERIOD, 64);
 
 	/* Init TX descriptors */
 	ste_init_tx_list(sc);
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20040330/e8420418/attachment.bin