Sudden mbuf demand increase and shortage under the load

[Sergey Babkin]

Maxim Sobolev wrote:
> 
> Hi,
> 
> Our company have a FreeBSD based product that consists of the numerous
> interconnected processes and it does some high-PPS UDP processing
> (30-50K PPS is not uncommon). We are seeing some strange periodic
> failures under the load in several such systems, which usually evidences
> itself in IPC (even through unix domain sockets) suddenly either
> breaking down or pausing and restoring only some time later (like 5-10
> minutes). The only sign of failure I managed to find was the increase of
> the "requests for mbufs denied" in the netstat -m and number of total
> mbuf clusters (nmbclusters) raising up to the limit.

As a simple idea: UDP is not flow-controlled. So potentially
nothing stops an application from sending the packets as fast 
as it can. If it's faster than the network card can process,
they would start collecting. So this might be worth a try
as a way to reproduce the problem and see if the system has
a safeguard against it or not.

Another possibility: what happens if a process is bound to
an UDP socket but doesn't actually read the data from it?
FreeBSD used to be pretty good at it, just throwing away
the data beyond a certain limit, SVR4 was running out of
network memory. But it might have changed, so might be
worth a look too.

-SB