A problem on TCP in High RTT Environment.

[Michael Tuexen]

On 11 Aug 2014, at 14:12, Eggert, Lars <lars at netapp.com> wrote:

> On 2014-8-11, at 9:17, Michael Tuexen <Michael.Tuexen at lurchi.franken.de> wrote:
>> Was there any suspicious output provided by netstat -m when the problems occur?
> 
> root at laurel:~ # netstat -m
> 8186/2179/10365 mbufs in use (current/cache/total)
> 8184/1214/9398/2036224 mbuf clusters in use (current/cache/total/max)
> 8184/885 mbuf+clusters out of packet secondary zone in use (current/cache)
> 0/5/5/1018111 4k (page size) jumbo clusters in use (current/cache/total/max)
> 0/0/0/301662 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/169685 16k jumbo clusters in use (current/cache/total/max)
> 18414K/2992K/21407K bytes allocated to network (current/cache/total)
> 544/57/8194 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
I guess the above is the problem. The card wants a lot of mbufs...
So the problem should go away if you increase the number of mbufs/clusters,
which means no requests are denied and you don't experience any performance
issue.
I ran into this on machines having several igb and ixgbe cards, each wanting
a lot of clusters for each of the receive rings per receive queue...

Best regards
Michael
> 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 0 requests for I/O initiated by sendfile
> 
> root at laurel:~ # uptime
> 2:12PM  up 37 mins, 3 users, load averages: 0.20, 0.25, 0.15
> 
> Lars