Kernel (7.3) crash due to mbuf leak?

[David DeSimone]

I will try asking this question again:

Can anyone give me some pointers as to how I can analyze these
crashdumps, or my running system, to determine what network subsystem is
leaking these mbuf's?

Before I start sifting through C code, I was wondering, does the mbuf
subsystem have a built-in means of tracking which other subsystems are
requesting memory, so that perhaps a clever gdb script can build a
histogram of which subsystems are consuming large amounts of mbuf's?
This would give me a pointer in the right direction to start
investigating the cause of the leak.


David DeSimone <fox at verio.net> wrote:
>
> After upgrading a couple of our systems from 7.2-RELEASE to 7.3-RELEASE,
> we have started to see them running out of mbuf's and crashing every
> month or so.  The panic string is:
> 
>     kmem_malloc(16384): kmem_map too small: 335233024 total allocated
> 
> The actual panic signature (backtrace) shows a memory allocation failure
> occurring in the filesystem code, but I do not think that is where the
> problem lies.  Instead, it is clear to me that the system is slowly
> leaking mbuf's until there is no more kernel memory available, and the
> filesystem is just the innocent bystander asking for memory and failing
> to get it.
> 
> Here's some netstat -m output on a couple of crashes:
> 
>     fs0# netstat -m -M vmcore.0
> 
>     882167/2902/885069 mbufs in use (current/cache/total)
>     351/2041/2392/25600 mbuf clusters in use (current/cache/total/max)
>     351/1569 mbuf+clusters out of packet secondary zone in use (current/cache)
>     0/199/199/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
>     0/0/0/19200 9k jumbo clusters in use (current/cache/total/max)
>     0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
>     221249K/5603K/226853K bytes allocated to network (current/cache/total)
>     0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
>     0/0/0 requests for jumbo clusters denied (4k/9k/16k)
>     0 requests for sfbufs denied
>     0 requests for sfbufs delayed
>     0 requests for I/O initiated by sendfile
>     0 calls to protocol drain routines

-- 
David DeSimone == Network Admin == fox at verio.net
  "I don't like spinach, and I'm glad I don't, because if I
   liked it I'd eat it, and I just hate it." -- Clarence Darrow


This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free.  Thank you.