bce(4) - com_no_buffers (Again)

[Tom Judge]

On 09/23/2010 03:30 PM, David Christensen wrote:
>>> Failure to allocate a new buffer should cause the driver to
>>> drop the received frame and reuse the buffer, not lock up the
>>> system.  Are you seeing the lockup come from bce(4) or does
>>> it come from somewhere else due to the dropped data?
>>>
>>>
>>>       
>> The lockup is not from the NIC as such, the systems have the appearance
>> of locking up as home directories are on NFS and the user information
>> is
>> stored in a remote LDAP server.   When the system starts to drop frames
>> due to lack of 9k memory regions it tends to last for a few minutes
>> (when it is really bad) and stop all traffic into the system.  This
>> appears to the average user as a complete system pause.
>>
>>     
> Ack.
>
>   
>> I am under the impression that the best solution is to tune the RX ring
>> so that flow control can be disabled but I not sure I could do this.
>>
>>     
> Are small frames common in the actual failing case or are they 
> restricted to your test case?  At least one of the Broadcom
> Linux drivers addresses this issue by copying frames < 128 bytes
> to a standard size buffer, saving the jumbo buffer allocation 
> and DMA mapping operation.  If small frames are at the root of 
> your problem then I think this would be a better solution than 
> allocating more 9KB blocks by increasing the ring size. 
>
>   
The cause of the problem is openldap's slapd sending search results back
to the hosts.  When it transmits the results it sends each record as a
separate frame (I guess the socket has TCP_NODELAY set but I have not
investigated this.), some search return ~1700 results, this causes small
packet storms on the interface and system 'pauses' as the slapd hosts
TCP stack retransmits the dropped data in larger jumbo frames.  This
type of search will trigger about 12k of data on the wire to be dropped
by the firmware and be retransmitted in 2 6k frames.

For me the ideal solution would be to have the following configuration:

* Header splitting turned on to reduce the chance of denied 9k allocations
* Flow control turned off.
* Increased RX ring to be able to be able to handle a link saturated
with 200 byte (or smaller) frames.

All of our systems running bce(4) NIC's with jumbo's have header
splitting turned on to reduce/remove issues with jumbo allocation.  We
first noticed this issue in 6.2 and have had use header splitting with
all our bce(4) hardware since then.

Tom

-- 
TJU13-ARIN