git: 3394d4239b85 - main - cam: allocate CCBs from UMA for SCSI and ATA IO

Sat May 15 23:24:41 UTC 2021

On Sat, May 15, 2021 at 11:05:49AM +0000, Edward Tomasz Napierala wrote:
> The branch main has been updated by trasz:
> 
> URL: https://cgit.FreeBSD.org/src/commit/?id=3394d4239b85b5577845d9e6de4e97b18d3dba58
> 
> commit 3394d4239b85b5577845d9e6de4e97b18d3dba58
> Author:     Edward Tomasz Napierala <trasz at FreeBSD.org>
> AuthorDate: 2021-05-15 10:17:22 +0000
> Commit:     Edward Tomasz Napierala <trasz at FreeBSD.org>
> CommitDate: 2021-05-15 11:03:49 +0000
> 
>     cam: allocate CCBs from UMA for SCSI and ATA IO
>     
>     This patch makes it possible for CAM to use small CCBs allocated
>     from an periph-specific UMA zone instead of the usual, huge ones.
>     The end result is that CCBs issued via da(4) take 544B (size of
>     ccb_scsiio) instead of the usual 2kB (size of 'union ccb', ~1.5kB,
>     rounded up by malloc(9)).  For ATA it's 272B.  We waste less
>     memory, we avoid zeroing the unused 1kB, and it should be easier
>     to allocate those CCBs in low memory conditions.  It should also
>     be possible to use uma_zone_reserve(9) to improve behaviour
>     in low memory conditions even further.
>     
>     Note that this does not change the size, or the layout, of CCBs
>     as such.  CCBs get allocated in various different ways, in particular
>     on the stack, and I don't want to redo all that.  Instead, this
>     provides an opt-in mechanism for the periph to declare "my start()
>     callback is fine with receiving a CCB allocated from this UMA zone".
>     In other words, most of the code works exactly as it used to; the
>     change only happens to IOs issued by xpt_run_allockq(), which
>     is - conveniently - pretty much all that matters for performance.
>     
>     The reason for doing it this way is that it's pretty small, localized
>     change, and can be implemented gradually and iteratively: take a
>     periph, make sure its start() callback only casts the CCBs it takes
>     to a particular type of CCB, for example ccb_scsiio, and that it only
>     casts CCBs returned by cam_periph_getccb() to that type, then add UMA
>     zone for that size, and declare it safe to XPT.
>     
>     This is disabled by default.  Set 'kern.cam.ada.enable_uma_ccbs=1'
>     and 'kern.cam.da.enable_uma_ccbs=1' tunables to enable it.  Testing
>     is welcome; I will flip the default to enable in two weeks from now.
>     
>     Reviewed By:    imp
>     Sponsored by:   NetApp, Inc.
>     Sponsored by:   Klara, Inc.
>     Differential Revision:  https://reviews.freebsd.org/D28674
> ---
>  sys/cam/ata/ata_da.c    | 18 ++++++++++++++++++
>  sys/cam/ata/ata_xpt.c   |  7 +++++++
>  sys/cam/cam_ccb.h       | 14 +++++++++++++-
>  sys/cam/cam_periph.h    |  3 +++
>  sys/cam/cam_xpt.c       | 32 +++++++++++++++++++++++++++++---
>  sys/cam/scsi/scsi_da.c  | 18 ++++++++++++++++++
>  sys/cam/scsi/scsi_xpt.c |  7 +++++++
>  7 files changed, 95 insertions(+), 4 deletions(-)

Hey Edward,

I suspect this may be the cause of a kernel panic I just had when
booting my Softiron Overdrive 1000 arm64 system:

panic: ata_action: ccb 0xffff000040388828, func_code 0x15 should not be allocated from UMA zone

cpuid = 1
time = 1
__HardenedBSD_version = 1400000 __FreeBSD_version = 1400013
version = FreeBSD 14.0-CURRENT-HBSD #1  hardened/current/master-n190374-a541810e5fa: Sat May 15 17:05:42 EDT 2021
    shawn at arm64-02:/usr/obj/usr/src/arm64.aarch64/sys/HARDENEDBSD
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x30
vpanic() at vpanic+0x188
panic() at panic+0x44
ata_action() at ata_action+0x340
_ata_announce_periph() at _ata_announce_periph+0x74
ata_announce_periph_sbuf() at ata_announce_periph_sbuf+0x24
xpt_announce_periph_sbuf() at xpt_announce_periph_sbuf+0x1a8
adaregister() at adaregister+0x420
cam_periph_alloc() at cam_periph_alloc+0x528
adaasync() at adaasync+0xa8
xptsetasyncfunc() at xptsetasyncfunc+0x100
xptdevicetraverse() at xptdevicetraverse+0x9c
xpttargettraverse() at xpttargettraverse+0x78
xptbustraverse() at xptbustraverse+0x80
xpt_register_async() at xpt_register_async+0x1fc
adainit() at adainit+0x4c
periphdriver_init() at periphdriver_init+0x54
xpt_finishconfig_task() at xpt_finishconfig_task+0x18
taskqueue_run_locked() at taskqueue_run_locked+0x188
taskqueue_thread_loop() at taskqueue_thread_loop+0x9c
fork_exit() at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x14
KDB: enter: panic
[ thread pid 0 tid 100023 ]
Stopped at      kdb_enter+0x44: undefined       f904411f

Thanks,

-- 
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/dev-commits-src-all/attachments/20210515/ac840bf1/attachment.sig>