git: 3394d4239b85 - main - cam: allocate CCBs from UMA for SCSI and ATA IO

[Gary Jennejohn]

On Sat, 15 May 2021 11:05:49 GMT
Edward Tomasz Napierala <trasz at FreeBSD.org> wrote:

> The branch main has been updated by trasz:
> 
> URL: https://cgit.FreeBSD.org/src/commit/?id=3394d4239b85b5577845d9e6de4e97b18d3dba58
> 
> commit 3394d4239b85b5577845d9e6de4e97b18d3dba58
> Author:     Edward Tomasz Napierala <trasz at FreeBSD.org>
> AuthorDate: 2021-05-15 10:17:22 +0000
> Commit:     Edward Tomasz Napierala <trasz at FreeBSD.org>
> CommitDate: 2021-05-15 11:03:49 +0000
> 
>     cam: allocate CCBs from UMA for SCSI and ATA IO
>     
>     This patch makes it possible for CAM to use small CCBs allocated
>     from an periph-specific UMA zone instead of the usual, huge ones.
>     The end result is that CCBs issued via da(4) take 544B (size of
>     ccb_scsiio) instead of the usual 2kB (size of 'union ccb', ~1.5kB,
>     rounded up by malloc(9)).  For ATA it's 272B.  We waste less
>     memory, we avoid zeroing the unused 1kB, and it should be easier
>     to allocate those CCBs in low memory conditions.  It should also
>     be possible to use uma_zone_reserve(9) to improve behaviour
>     in low memory conditions even further.
>     
>     Note that this does not change the size, or the layout, of CCBs
>     as such.  CCBs get allocated in various different ways, in particular
>     on the stack, and I don't want to redo all that.  Instead, this
>     provides an opt-in mechanism for the periph to declare "my start()
>     callback is fine with receiving a CCB allocated from this UMA zone".
>     In other words, most of the code works exactly as it used to; the
>     change only happens to IOs issued by xpt_run_allockq(), which
>     is - conveniently - pretty much all that matters for performance.
>     
>     The reason for doing it this way is that it's pretty small, localized
>     change, and can be implemented gradually and iteratively: take a
>     periph, make sure its start() callback only casts the CCBs it takes
>     to a particular type of CCB, for example ccb_scsiio, and that it only
>     casts CCBs returned by cam_periph_getccb() to that type, then add UMA
>     zone for that size, and declare it safe to XPT.
>     
>     This is disabled by default.  Set 'kern.cam.ada.enable_uma_ccbs=1'
>     and 'kern.cam.da.enable_uma_ccbs=1' tunables to enable it.  Testing
>     is welcome; I will flip the default to enable in two weeks from now.
>     

[snip diff]

I built and installed world and kernel and enabled both tunables.

System's been up for about two hours using both ada and da disks
pretty intensely and hasn't trashed any yet.

-- 
Gary Jennejohn