Multiple netgraph threads

[Robert Watson]

On Sun, 30 Mar 2008, Alexander Motin wrote:

> I have implemented a patch (for the HEAD) making netgraph to use several own 
> threads for event queues processing instead of using single swinet. It 
> should significantly improve netgraph SMP scalability on complicated 
> workloads that require queueing by implementation (PPTP/L2TP) or stack size 
> limitations. It works perfectly on my UP system, showing results close to 
> original or even a bit better. I have no real SMP test server to measure 
> real scalability, but test on HTT CPU works fine, utilizing both virtual 
> cores at the predictable level. Reviews and feedbacks are welcome. URL: 
> http://people.freebsd.org/~mav/netgraph.threads.patch

FYI, you might be interested in some similar work I've been doing in the 
rwatson_netisr branch in Perforce, which:

(1) Adds per-CPU netisr threads
(2) Introduces inpcb affinity, assigned using a hash on the tuple (similar to
     RSS) to load balance
(3) Moves to rwlock use on inpcb and pcbinfo, used extensively in UDP and
     somewhat in TCP

My initial leaning would be that we would like to avoid adding too many more 
threads that will do per-packet work, as that leads to excessive context 
switching.  I have similar worries regarding ithreads, and I suspect (hope?) 
we will have an active discussion of this at the BSDCan developer summit.

BTW, I'd be careful with the term "should" and SMP -- often times scalability 
to multiple cores involves not just introducing the opportunity for 
parallelism, but also significantly refining or changing the data model to 
allow that parallelism to be efficiently used.  Right now, despite loopback 
performance being a bottleneck with a single netisr thread, we're not seeing a 
performance improvement for database workloads over loopback with multiple 
netisr threads.  We're diagnosing this still -- initially it appeared to be 
tcbinfo lock contention (not surprising), but moving to read locking on 
tcbinfo didn't appear to help (except in reducing contention leading to more 
idle time rather than more progress).  The current theory is that something 
about the approach isn't working well with the scheduler but we need to 
analyze the scheduler traces further.  My recommendation would be that you do 
a fairly thorough performance evaluation before committing.

Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
freebsd-hackers at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"