l2arc_feed_thread cpu utlization

[Andriy Gapon]

This is just a heads up, no patch yet.

l2arc_feed_thread periodically wakes up and scans certain amount of ARC buffers
and writes eligible buffers to a cache device.
Number of scanned buffers is limited by a threshold on the amount of data in the
buffers seen.  The threshold is applied on a per buffer list basis.  In upstream
there are 4 relevant lists: (data, metadata) X (MFU, MRU).  In FreeBSD each of
the lists was subdivided into 16 lists.  This was done to reduce contention on
the locks that protect the lists.  But as a side effect l2arc_feed_thread can
scan 16 times more data (~ buffers).

So, if you have a rather large ARC and L2ARC and your buffers tend to be
sufficiently small, then you could observe l2arc_feed_thread burning a
noticeable amount of CPU.  On some of our systems I observed it using up to 40%
of a single core.  Scaling back the threshold by factor of 16 makes CPU
utilization go down by the same factor.

I plan to commit this change to FreeBSD ZFS code.
Any comments are welcome.
-- 
Andriy Gapon