[Fwd: Re: bge Ierr rate increase from 5.3R -> 6.1R]

[David Christensen]

> These happen under loads that can't be handled, and generally cause
> thousands of input errors every second.  The hardware records dropped
> packets separately from other input errors, but unfortunately 
> all types
> of input errors are counted together in if_ierrors, and I haven't done
> more than muck around in ddb to separate them.

There are a couple places I can suggest you look to understand if there
are any resource limitations being encountered by the controller.  The
first is register 0x4450 (bits 15:0 for 5700 to 5704 and bits 8:0 for
other controllers) which shows the number of internal 
free RX MBUFs on the controller.  (In this case an MBUF is an internal 
128 byte data structure used by the controller.)  This value is
especially 
important on 5700 to 5704 devices since those devices share a pool of 
MBUFs between the RX and TX data paths, unlike the 5705 and later
devices
which use separate MBUF pools for RX and TX data.  As the value
approaches 
0, the controller will either start to generate pause frames or simply 
drop packets as specified by the values in registers 0x4410, 0x4414, and

0x4418. If you see ifInDiscards incrementing then this is a definite
possibility.  You would also see bit 4 of register 0x4400 set if this
occurs.

I also noticed on -CURRENT that register 0x2c18 was set to a rather high
value (0x40).  This register controls how many bge_rx_bd's will be
fetched at
one time.  A high value will reduce PCI bus utilization as it will fetch
more BD's in a single transaction, but it will increase latency and
could
potentially deplete the available internal MBUFs as the controller waits
for the driver to populate additional bge_rx_bd's.  Try using a smaller 
value such as 0x08.

It would be really interesting if you could add a sysctl that would
bring
out the hardware statistics maintained by the device, similar to what
exists
in the "bce" driver.  With this information we could focus on individual
blocks to see where the packet loss is occurring and may be able to come
up with more ideas on tuning the controller.

> bge also has a "mini" rx ring which FreeBSD doesn't use.  I 
> don't really
> understand this or the interaction of the separate rx rings, but hope
> that the mini ring can be used to handle small packets and would only
> need an mbuf (not an mbuf cluster) for each packet, and with 
> it and the
> jumbo ring the total hardware buffering would be 1024(mini) + 
> 512(std) +
> 256(jumbo), with the 1024-entry event ring only used to communicate
> with the host and its size not really limiting buffering.  Meeting a
> latency requirement of 1024+512 tiny-packet times is much easier than
> meeting one of 192 or 20 tiny-packet times.  (I only actually saw the
> limits of 20 and 192 for full-sized (non jumbo) packets.)

Mini rings are only supported by 57XX controllers that support external
SSRAM, and the only controller that supports that is the 5700.  This 
feature was never used in production so I wouldn't consider it an option
since you'll never find the right hardware to support it.

Dave