FreeBSD 4.9 losing mbufs!!!

[Stephen Clark]

Robert Watson wrote:

>On Wed, 26 Apr 2006, Stephen Clark wrote:
>
>  
>
>>>Sorry not to have caught this thread earlier; I've been on travel for the 
>>>last few weeks.  My general suggestion would be to try to narrow the code 
>>>paths traversed to try to eliminate as much code as possible from the 
>>>search.  It sounds like you've done that pretty effectively :-).
>>>
>>>Typically, memory leaks occur in edge error cases, where the memory is not 
>>>properly released, or ownership is unclear.  My suggestion would be to add 
>>>counters (or look at existing counters where already present) and see if 
>>>there's an error case being triggered in about the same quantity that mbuf 
>>>leakage is occuring.  Chances are, there's an error being returned and a 
>>>missing m_freem().
>>>
>>>Based on your comments above, I might also pay attention to the routing 
>>>socket path -- the rate of leak could correspond to the routing daemons 
>>>talking to the network stack, rather than the rate of traffic.  For 
>>>example, it could be that one of the routing messages is handled improperly 
>>>resulting in a leak.
>>>
>>>Unfortunately, tracking down memory leaks can be quite difficult, and tends 
>>>to require a combination of dogged persistence and luck...
>>>      
>>>
>>Good news and bad news.
>>
>>I managed to get enough of our system running on 6.x stable to test and it 
>>does not appear to lose mbufs. Bad news my ipsec transfer rate dropped from 
>>54mbits/sec to 39mbits/sec. We need to be able to handle a t3 (45mbits/sec). 
>>Any ideas as to why this drop off in 6.x?
>>    
>>
>
>Well, that's good about the good news, but not so great about the bad news. 
>Are you using IPv6?  If not, could you look at how FAST_IPSEC performs instead 
>of the default IPSEC package, if you're not already doing so?  The KAME IPSEC 
>implementation is not multi-processor compatible, so when IPSEC support is 
>compiled into the kernel, execution of the network stack and related 
>components is limited to a single CPU (you'll see a warning about this at boot 
>if this is the case).  This is something we hope to fix in a future release, 
>but in the mean time FAST_IPSEC offers improved performance and SMP 
>scalability, as well as support for hardware crypto acceleration.  The 
>downside to FAST_IPSEC is that it doesn't currently support IPv6, which is 
>also something we'd like to fix in the future.
>
>Aside from the above, there are a number of other things we can look at to 
>decide what the source of the performance problem is.  First, I'd encourage 
>you to make sure any system debugging features, such as INVARIANTS, 
>INVARIANT_SUPPORT, and WITNESS, are disabled.  Next, I would recommend 
>generating a high level of your load on the system, and using systat -vmstat 1 
>and top -S to grab some information about it in the steady state.  I recommend 
>letting both run for a couple of minutes, then grabbing the output from both 
>of them.  This will give us an idea of where CPU usage is going in the kernel, 
>which is where I assume most of your workload ends up being handled.
>
>Thanks,
>
>Robert N M Watson
>_______________________________________________
>freebsd-stable at freebsd.org mailing list
>http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>
>  
>
Thanks Robert,

I'll give that a try.

Regards,
Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)