[zone: Mbuf_cluster] kern.ipc.nmbclusters limit reached in Virtual machine causes downtime

[Alan Somers]

On Tue, Jun 24, 2014 at 2:53 AM, Stefan Stere <icetor.is at bitmessage.ro> wrote:
> Hi,
>
> I have a vmware virtual server running FreeBSD 10.0 STABLE
> The virtual server has 100mbps port.
>
> It is running a Tor router, consuming an average of 6-7 TB of monthly traffic. Its the only purpose of the server.
>
> Last night it went down, and vmware console log was saying:
> [zone: Mbuf_cluster] kern.ipc.nmbclusters limit reached
>
>
> I don't know what this means - the traffic of the server is unlimited and nothing is capped in any way. What can I do to fix this? I have read on freebsd wiki that I might need to add some lines to sysctl ? can you please confirm? Thank you in advance.

This might be related to a problem that Alon Ronen discovered.  The
kernel can leak mbufs when experiencing memory pressure, if you're
using SOCK_DGRAM or SOCK_SEQPACKET Unix domain sockets (or even
SOCK_STREAM if sending ancillary data).  You could try the attached
patch that Alon and I are working on.

Even if the patch doesn't fix your problem, it would be interesting to
see the output of "vmstat -z".

-Alan
-------------- next part --------------
Index: sys/kern/uipc_usrreq.c
===================================================================
--- sys/kern/uipc_usrreq.c	(revision 267818)
+++ sys/kern/uipc_usrreq.c	(working copy)
@@ -970,10 +970,15 @@
 		case SOCK_STREAM:
 			if (control != NULL) {
 				if (sbappendcontrol_locked(&so2->so_rcv, m,
-				    control))
+				    control)) {
 					control = NULL;
-			} else
+					m = NULL;
+				} else
+					error = ENOBUFS;
+			} else {
 				sbappend_locked(&so2->so_rcv, m);
+				m = NULL;
+			}
 			break;
 
 		case SOCK_SEQPACKET: {
@@ -987,8 +992,11 @@
 			 * level up the stack.
 			 */
 			if (sbappendaddr_nospacecheck_locked(&so2->so_rcv,
-				from, m, control))
+				from, m, control)) {
 				control = NULL;
+				m = NULL;
+			} else
+				error = ENOBUFS;
 			break;
 			}
 		}
@@ -1009,7 +1017,6 @@
 			so->so_snd.sb_flags |= SB_STOP;
 		SOCKBUF_UNLOCK(&so->so_snd);
 		UNP_PCB_UNLOCK(unp2);
-		m = NULL;
 		break;
 
 	default: