New CAM locking preview

[Alexander Motin]

On 16.08.2013 04:12, Adrian Chadd wrote:
> Cool!
>
> I assume you've run this with full witness debugging enabled, to catch
> lock ordering issues?

Of course! I've endless times switched between debug and normal builds 
to test both correctness and performance after each change. But more 
external tests are welcome.

> This is great. I look forward to per-CPU, pinned, completion threads
> that I can do interesting things with (like schedule aio-sendfile
> completions..)
>
> On 15 August 2013 14:40, Alexander Motin <mav at freebsd.org
> <mailto:mav at freebsd.org>> wrote:
>
>     Hi.
>
>     Last weeks I've made substantial progress on my CAM locking work. In
>     fact, at this moment I think I've tied all loose ends good enough to
>     consider the new design viable and implementation worth further
>     testing and bug fixing. So I would like to ask for review of my work
>     from everybody who interested in CAM internals.
>
>     In short, my idea was to split single per-SIM lock, that creates
>     huge congestion under high IOPS, into several smaller ones. So
>     design I've finally chosen includes such locks:
>       1) New per-device (per-LUN) locks to protect state of the devices
>     and respective periphs. In most cases peripheral drivers just use
>     that lock instead of SIM lock used before, so code modification is
>     minimal and straightforward.
>       2) New per-target lock to protect list of LUNs fetched from the
>     device.
>       3) Old single per-SIM lock to protect SIM driver internals, but
>     only that. No parts of CAM itself use that lock. Keeping it for SIMs
>     allows to keep API and hopefully ABI compatibility. Reducing its
>     scope allows to reduce congestion.
>       4) New per-SIM lock to protect SIM and device command queues. That
>     allows execute queued commands from any context unrelated to other
>     locks. Also this lock serializes accesses to sim_action() method for
>     the most of commands, this allows to mostly avoid busy spilling on
>     SIM lock collision.
>       5) New per-bus locks to protect target, device and periphs
>     reference counters. It allows to create and destroy paths unrelated
>     to other locks in any possible context.
>
>     Numbers above also define supposed lock ordering: while holding
>     per-device lock 1) is allowed to request SIM lock 3), but not
>     backward. Cases where opposite is required (command completions and
>     async events) are handled via queuing events via several completion
>     threads. The rest of locks are self-contained and does not really
>     suppose cascading.
>
>     All these changes combined with GEOM direct dispatch (it will be
>     next separate project) allow to double system performance in disk
>     I/O microbenchmarks, comparing to present head, same as it was
>     announced on 2013-05 DevSummit:
>     http://people.freebsd.org/~__mav/camlock.pdf
>     <http://people.freebsd.org/~mav/camlock.pdf> . Tests without GEOM
>     changes also show performance improvement, but limited by heavy
>     bottleneck at the GEOM g_up/g_down threads at the level of 5-20%.
>
>     Project sources could be found at SVN projects/camlock branch:
>     http://svnweb.freebsd.org/__base/projects/camlock/
>     <http://svnweb.freebsd.org/base/projects/camlock/> . Many early
>     changes from that branch are already integrated to head, so to
>     simplify review the rest patches for changes before r254059 were
>     manually remade and could be found here:
>     http://people.freebsd.org/~__mav/camlock_patches/
>     <http://people.freebsd.org/~mav/camlock_patches/> .
>
>     These changes do not require controller driver modifications,
>     keeping KPIs and hopefully KBIs intact, but create base for later
>     work to use multiqueue capabilities of new controllers.
>
>     This work is sponsored by iXsystems, Inc.


-- 
Alexander Motin