arm pmap locking

[Ian Lepore]

On Sun, 2012-09-23 at 12:10 -0500, Alan Cox wrote:
> On 09/23/2012 09:30, Ian Lepore wrote:
> > On Sat, 2012-09-22 at 19:55 -0500, Alan Cox wrote:
> >> On 09/22/2012 15:45, Ian Lepore wrote:
> >>> Yep.  Looks like the first one (reversed) is very very early in the
> >>> init.  After this one, there were no other LORs during the boot until
> >>> the one that you said is fixed now in -current.
> >>>
> >>> lock order reversal:
> >>>    1st 0xc055bd08 4096 (UMA zone) @ /usr/src/sys/vm/uma_core.c:2011
> >>>    2nd 0xc043c198 pmap (pmap) @ /usr/src/sys/arm/arm/pmap.c:3734
> >> Ah.  Notice that this is an "UMA zone" lock, but nonetheless a different
> >> zone and lock from before, i.e., "4096" versus "PV ENTRY".  By default,
> >> witness treats all uma zone locks as having the same lock ordering.
> >> This helps to keep the distinct lock types that witness must deal with
> >> and consequently its run-time overhead down.  However, sometimes uma
> >> zone locks on zones that are used within the VM system need to be
> >> handled specially.  There is an argument to uma_zcreate() for doing
> >> this: UMA_ZONE_MTXCLASS
> >>
> >> Try this now.  Remove whatever change that you made to witness and apply
> >> a change like this:
> >>
> >> Index: arm/arm/pmap.c
> >> ===================================================================
> >> --- arm/arm/pmap.c      (revision 240803)
> >> +++ arm/arm/pmap.c      (working copy)
> >> @@ -1821,7 +1821,7 @@ pmap_init(void)
> >>            * Initialize the PV entry allocator.
> >>            */
> >>           pvzone = uma_zcreate("PV ENTRY", sizeof (struct pv_entry),
> >> NULL, NULL,
> >> -           NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM | UMA_ZONE_NOFREE);
> >> +           NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_MTXCLASS | UMA_ZONE_VM);
> >>           TUNABLE_INT_FETCH("vm.pmap.shpgperproc",&shpgperproc);
> >>           pv_entry_max = shpgperproc * maxproc + cnt.v_page_count;
> >>           uma_zone_set_obj(pvzone,&pvzone_obj, pv_entry_max);
> >>
> >> I've also removed UMA_ZONE_NOFREE from the arguments because it's
> >> redundant.  uma_zone_set_obj() itself sets that flag.
> >>
> >> Alan
> >>
> > Did that, now the original LOR that happens at start_init() time
> > involves pmap and the malloc 4096 uma zone rather than the PV ENTRY
> > zone.  Putting back in the preset order, it looks like a malloc() in
> > mtx_pool_create() is now the first call with the wrong order, the bottom
> > of the call stack looks like this:
> >
> > malloc() at malloc+0x10
> > scp=0xc00afe98 rlv=0xc00b28c8 (mtx_pool_create+0x54)
> >          rsp=0xc04ddeb4 rfp=0xc04dded0
> >          r10=0x204fe888 r9=0x00000019
> >          r8=0x20479a10 r7=0x00000000 r6=0xc0254a10 r5=0x00000080
> >          r4=0xc0276a40
> > mtx_pool_create() at mtx_pool_create+0x10
> > scp=0xc00b2884 rlv=0xc00b290c (mtx_pool_setup_dynamic+0x1c)
> >          rsp=0xc04dded4 rfp=0xc04ddee0
> >          r7=0x20000124 r6=0x00000004
> >          r5=0x20000130 r4=0xc0276a40
> > mtx_pool_setup_dynamic() at mtx_pool_setup_dynamic+0x10
> > scp=0xc00b2900 rlv=0xc0082654 (mi_startup+0xf8)
> >          rsp=0xc04ddee4 rfp=0xc04ddef4
> > mi_startup() at mi_startup+0x10
> > scp=0xc008256c rlv=0xc00001cc (virt_done+0x18)
> >          rsp=0xc04ddef8 rfp=0x00000000
> >          r4=0x2000020c
> >
> > If you need complete backtraces or other details let me know.
> >
> 
> Thanks.  I'd like to see the stack trace without any preset lock order.
> 
> Unless I explicitly say otherwise, assume that I want UMA_ZONE_MTXCLASS 
> specified to the pvzone creation.
> 
> In general, uma zone locks should come before the pmap lock in the 
> expected lock order.  However, any of the zones used by the pmap are 
> likely to be exceptions to this rule.  After you added UMA_ZONE_MTXCLASS 
> to the PV zone, I was expecting the next reported LOR to be on the l2 
> zone or l2table zone.  Does that make sense?
> 
> Alan
> 

Without presetting the order...

warning: no time-of-day clock registered, system time will not be set
accurately
lock order reversal:
 1st 0xc0a8a0b0 pmap (pmap) @ /usr/src/sys/arm/arm/pmap.c:971
 2nd 0xc055b388 128 Bucket (UMA zone) @ /usr/src/sys/vm/uma_core.c:2485
KDB: stack backtrace:
db_trace_thread() at db_trace_thread+0x10
scp=0xc0215d7c rlv=0xc0215f80 (db_trace_self+0x1c)
        rsp=0xc2e9d90c rfp=0xc2e9d918
        r10=0x00000000 r9=0xc0a4c4b8
        r8=0xc040aec4 r7=0xffffffff r6=0xc0a4c6c0 r5=0xc023a21c
        r4=0xc2e9d924
db_trace_self() at db_trace_self+0x10
scp=0xc0215f74 rlv=0xc001ee0c (db_trace_self_wrapper+0x30)
        rsp=0xc2e9d91c rfp=0xc2e9da38
db_trace_self_wrapper() at db_trace_self_wrapper+0x10
scp=0xc001edec rlv=0xc00ef4f8 (kdb_backtrace+0x3c)
        rsp=0xc2e9da3c rfp=0xc2e9da4c
        r4=0xc02dbf44
kdb_backtrace() at kdb_backtrace+0x10
scp=0xc00ef4cc rlv=0xc0102c48 (_witness_debugger+0x2c)
        rsp=0xc2e9da50 rfp=0xc2e9da64
        r4=0x00000001
_witness_debugger() at _witness_debugger+0x10
scp=0xc0102c2c rlv=0xc0103468 (witness_checkorder+0x7e8)
        rsp=0xc2e9da68 rfp=0xc2e9dab4
        r5=0x00000000 r4=0xc055b388
witness_checkorder() at witness_checkorder+0x10
scp=0xc0102c90 rlv=0xc00b37fc (_mtx_lock_flags+0xa4)
        rsp=0xc2e9dab8 rfp=0xc2e9dadc
        r10=0x00000000 r9=0xc043c5ac
        r8=0x00000000 r7=0x000009b5 r6=0xc026e6b4 r5=0x00000000
        r4=0xc055b388
_mtx_lock_flags() at _mtx_lock_flags+0x10
scp=0xc00b3768 rlv=0xc01f2e20 (zone_alloc_item+0x34)
        rsp=0xc2e9dae0 rfp=0xc2e9db08
        r8=0xc026e6b4 r7=0x00000201
        r6=0x00000000 r5=0xc055a3c0 r4=0x00000201
zone_alloc_item() at zone_alloc_item+0x10
scp=0xc01f2dfc rlv=0xc01f46a4 (bucket_alloc+0x3c)
        rsp=0xc2e9db0c rfp=0xc2e9db20
        r10=0x00000000 r8=0xc055a6e0
        r7=0xc055a740 r6=0x00000000 r5=0xc02cd428 r4=0x00000201
bucket_alloc() at bucket_alloc+0x10
scp=0xc01f4678 rlv=0xc01f4f8c (uma_zalloc_arg+0x450)
        rsp=0xc2e9db24 rfp=0xc2e9db60
        r5=0x00000201 r4=0xc026e6b4
uma_zalloc_arg() at uma_zalloc_arg+0x10
scp=0xc01f4b4c rlv=0xc021b4d0 (pmap_get_pv_entry+0x40)
        rsp=0xc2e9db64 rfp=0xc2e9db70
        r10=0x00000000 r9=0xc043c5ac
        r8=0x2078e55e r7=0xc0a8a0b0 r6=0xc0272f84 r5=0xc05370b8
        r4=0x00000000
pmap_get_pv_entry() at pmap_get_pv_entry+0x10
scp=0xc021b4a0 rlv=0xc02201a0 (pmap_enter_locked+0xcfc)
        rsp=0xc2e9db74 rfp=0xc2e9dbf0
pmap_enter_locked() at pmap_enter_locked+0x10
scp=0xc021f4b4 rlv=0xc0220704 (pmap_enter+0x74)
        rsp=0xc2e9dbf4 rfp=0xc2e9dc20
        r10=0x00000007 r9=0xc2e9dde4
        r8=0xc056dea0 r7=0xbffff000 r6=0xc043a07c r5=0xc0a8a0b0
        r4=0xc0272f84
pmap_enter() at pmap_enter+0x10
scp=0xc02206a0 rlv=0xc01f80e0 (vm_fault+0x1688)
        rsp=0xc2e9dc24 rfp=0xc2e9dd3c
        r10=0x00000002 r8=0xbffff000
        r7=0xc0a88000 r6=0x00000000 r5=0xc0273420 r4=0xc056dea0
vm_fault() at vm_fault+0x10
scp=0xc01f6a68 rlv=0xc0224078 (data_abort_handler+0x360)
        rsp=0xc2e9dd40 rfp=0xc2e9dde0
        r10=0x00000002 r9=0xc2e9dde4
        r8=0xbffff000 r7=0xc0a88000 r6=0x00000000 r5=0xc0273420
        r4=0xc0a86088
data_abort_handler() at data_abort_handler+0x10
scp=0xc0223d28 rlv=0xc0217784 (exception_exit)
        rsp=0xc2e9dde4 rfp=0xc2e9de84
        r10=0xc02b7d0a r9=0xc0a86000
        r8=0xc2e9deac r7=0xc02b7d0a r6=0xc02b7d00 r5=0xffff1004
        r4=0xffffffff
start_init() at start_init+0x10
scp=0xc0082244 rlv=0xc009c460 (fork_exit+0x84)
        rsp=0xc2e9de88 rfp=0xc2e9dea8
        r10=0x00000000 r9=0x00000000
        r8=0xc2e9deac r7=0x00000000 r6=0xc0082234 r5=0xc0a86000
        r4=0xc0a88000
fork_exit() at fork_exit+0x10
scp=0xc009c3ec rlv=0xc022354c (fork_trampoline+0x14)
        rsp=0xc2e9deac rfp=0x00000000
        r8=0x00000000 r7=0x6c6c6568
        r6=0x735f6365 r5=0x00000000 r4=0xc0082234

-- Ian