ARM pmap/UMA LOR in init(8)

[Rafal Jaworowski]

Hi,

We have identified the following lock order reversal that shows up early in
the init(8); after closer look it seems harmless, so the description below is
mostly for the record when it scares other people as well...

[credits for the investigation and analysis go to Grzegorz]

1. It occurs only when *both* WITNESS and INVARIANTS are set in kernel config.

lock order reversal:

 1st 0xc195108c pmap (pmap) @ src/sys/arm/arm/pmap.c:971
 2nd 0xc0f1b608 PV ENTRY (UMA zone) @ src/sys/vm/uma_core.c

backtrace:
witness_checkorder() at witness_checkorder+0x10
scp=0xc04abb48 rlv=0xc046bd88 (_mtx_lock_flags+0x60)
        rsp=0xc7882b0c rfp=0xc7882b34
        r10=0xc0e6efd8 r9=0x00000001
        r8=0x00000752 r7=0xc05b9d28 r6=0x00000000 r5=0xc0f1b608
        r4=0xc0df8988
_mtx_lock_flags() at _mtx_lock_flags+0x10
scp=0xc046bd38 rlv=0xc05537fc (uma_zalloc_arg+0x94)
        rsp=0xc7882b38 rfp=0xc7882b78
        r10=0x0141f55e r8=0xc0f1a580
        r7=0xc0f1a5c8 r6=0x00000000 r5=0xc0f030b8 r4=0x0141f552
uma_zalloc_arg() at uma_zalloc_arg+0x10
scp=0xc0553778 rlv=0xc057b3a0 (pmap_release+0x112c)
        rsp=0xc7882b7c rfp=0xc7882be0
        r10=0x0141f55e r9=0x00000001
        r8=0x00000000 r7=0xc195108c r6=0x00000000 r5=0xc0f030b8
        r4=0x0141f552
pmap_release() at pmap_release+0x2d4
scp=0xc057a548 rlv=0xc057b838 (pmap_enter+0x74)
        rsp=0xc7882be4 rfp=0xc7882c14
        r10=0xc0f8baa8 r9=0xc0e655ac
        r8=0xbffff000 r7=0x00000000 r6=0xc195108c r5=0xc05be584
        r4=0x00000007
pmap_enter() at pmap_enter+0x10
scp=0xc057b7d4 rlv=0xc0556d50 (vm_fault+0x1438)
        rsp=0xc7882c18 rfp=0xc7882d34
        r10=0xc7882ddc r9=0xc7882ef8
        r8=0x00000002 r7=0xc194a440 r6=0x00000000 r5=0x00000001
        r4=0xc0f8baa8
vm_fault() at vm_fault+0x10
scp=0xc0555928 rlv=0xc058048c (data_abort_handler+0x1ec)
        rsp=0xc7882d38 rfp=0xc7882dd8
        r10=0xc7882ddc r9=0xc7882ef8
        r8=0x00000002 r7=0xc194a440 r6=0x00000000 r5=0xbffff000
        r4=0xc194d090
data_abort_handler() at data_abort_handler+0x10
scp=0xc05802b0 rlv=0xc0573fa8 (address_exception_entry+0x50)
        rsp=0xc7882ddc rfp=0xc7882e7c
        r10=0x00000000 r9=0xc194d000
        r8=0xc0df2a9c r7=0xc0df2aa6 r6=0xc044ac78 r5=0xe0000004
        r4=0xbfffffff
exec_shell_imgact() at exec_shell_imgact+0x614
scp=0xc044ac88 rlv=0xc045e5d0 (fork_exit+0x78)
        rsp=0xc7882e80 rfp=0xc7882ea8
        r10=0x00000000 r9=0x00000001
        r8=0xc7882eac r7=0xc194d000 r6=0xc044ac78 r5=0xc0e6efd8
        r4=0xc194a440
fork_exit() at fork_exit+0x10
scp=0xc045e568 rlv=0xc057fb28 (fork_trampoline+0x14)
        rsp=0xc7882eac rfp=0x00000000
        r10=0x00000000 r9=0x00000000
        r8=0x00000000 r7=0xc0558268 r6=0xc785aeac r5=0x00000000
        r4=0xc044ac78


2. From witness(4) perspective we have the follwing calls flow:

witness_checkorder()          <--- here pmap mutex is set as a child of 'UMA zone'
mtx_lock_flags()              <--- pmap mutex is taken
pmap_extract()
uma_dbg_getslab()
uma_dbg_alloc()               <--- it is called only with INVARIANTS
uma_zalloc_arg()              <--- here 'UMA zone' mutex is taken
malloc()
mtx_pool_create()
mtx_pool_setup_dynamic()
mi_startup()

witness_checkorder() sets pmap mutex as a child of 'UMA zone' mutex since 'UMA
zone' is already taken; so far all is considered correct by witness(4).


3. Here's an interesting part of ddb 'sh witness' trace (with pmap vs UMA zone
order):

0 devfsmount
1  devfs interlock
5   vnode interlock
6    cdev
6    vnode_free_list
6  vnode_free_list
2  UMA zone
5   pmap
5  vnode interlock

At this angle, pmap mutex is taken first and then we try to take UMA zone
mutex, witness(4) considers this an order violation and screams.

4. The problem is witness(4) uses lock type to differentiate between locking
objects. However, this field is hardly ever used when creating locking
objects, so most often the name is used instead. In effect, all mutexes for
pmap structures end up of the same 'pmap' type, similarly for UMA zone
mutexes: whereas each zone has physically separate locking object, they all
seem as one 'UMA zone' mutex...

In summary this LOR doesn't concern the same mutexes that witness(4) thinks,
but it's unable to discern this.

Rafal