b_freelist TAILQ/SLIST

[Alexander Motin]

Hi.

While doing some profiles of GEOM/CAM IOPS scalability, on some test 
patterns I've noticed serious congestion with spinning on global 
pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is 
already very simple, I've tried to optimize probably the only thing 
possible there: switch bswlist from TAILQ to SLIST. As I can see, 
b_freelist field of struct buf is really used as TAILQ in some other 
places, so I've just added another SLIST_ENTRY field. And result 
appeared to be surprising -- I can no longer reproduce the issue at all. 
May be it was just unlucky synchronization of specific test, but I've 
seen in on two different systems and rechecked results with/without 
patch three times.

The present patch is here:
http://people.freebsd.org/~mav/buf_slist.patch

The question is how to do it better? What is the KPI/KBI policy for 
struct buf? I could replace b_freelist by a union and keep KBI, but 
partially break KPI. Or I could add another field, probably breaking 
KBI, but keeping KPI. Or I could do something handmade with no breakage. 
Or this change is just a bad idea?

-- 
Alexander Motin