arm pmap locking

[Ian Lepore]

On Sat, 2012-09-22 at 19:55 -0500, Alan Cox wrote:
> On 09/22/2012 15:45, Ian Lepore wrote:
> > Yep.  Looks like the first one (reversed) is very very early in the
> > init.  After this one, there were no other LORs during the boot until
> > the one that you said is fixed now in -current.
> >
> > lock order reversal:
> >   1st 0xc055bd08 4096 (UMA zone) @ /usr/src/sys/vm/uma_core.c:2011
> >   2nd 0xc043c198 pmap (pmap) @ /usr/src/sys/arm/arm/pmap.c:3734
> 
> Ah.  Notice that this is an "UMA zone" lock, but nonetheless a different 
> zone and lock from before, i.e., "4096" versus "PV ENTRY".  By default, 
> witness treats all uma zone locks as having the same lock ordering.  
> This helps to keep the distinct lock types that witness must deal with 
> and consequently its run-time overhead down.  However, sometimes uma 
> zone locks on zones that are used within the VM system need to be 
> handled specially.  There is an argument to uma_zcreate() for doing 
> this: UMA_ZONE_MTXCLASS
> 
> Try this now.  Remove whatever change that you made to witness and apply 
> a change like this:
> 
> Index: arm/arm/pmap.c
> ===================================================================
> --- arm/arm/pmap.c      (revision 240803)
> +++ arm/arm/pmap.c      (working copy)
> @@ -1821,7 +1821,7 @@ pmap_init(void)
>           * Initialize the PV entry allocator.
>           */
>          pvzone = uma_zcreate("PV ENTRY", sizeof (struct pv_entry), 
> NULL, NULL,
> -           NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM | UMA_ZONE_NOFREE);
> +           NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_MTXCLASS | UMA_ZONE_VM);
>          TUNABLE_INT_FETCH("vm.pmap.shpgperproc", &shpgperproc);
>          pv_entry_max = shpgperproc * maxproc + cnt.v_page_count;
>          uma_zone_set_obj(pvzone, &pvzone_obj, pv_entry_max);
> 
> I've also removed UMA_ZONE_NOFREE from the arguments because it's 
> redundant.  uma_zone_set_obj() itself sets that flag.
> 
> Alan
> 

Did that, now the original LOR that happens at start_init() time
involves pmap and the malloc 4096 uma zone rather than the PV ENTRY
zone.  Putting back in the preset order, it looks like a malloc() in
mtx_pool_create() is now the first call with the wrong order, the bottom
of the call stack looks like this:

malloc() at malloc+0x10
scp=0xc00afe98 rlv=0xc00b28c8 (mtx_pool_create+0x54)
        rsp=0xc04ddeb4 rfp=0xc04dded0
        r10=0x204fe888 r9=0x00000019
        r8=0x20479a10 r7=0x00000000 r6=0xc0254a10 r5=0x00000080
        r4=0xc0276a40
mtx_pool_create() at mtx_pool_create+0x10
scp=0xc00b2884 rlv=0xc00b290c (mtx_pool_setup_dynamic+0x1c)
        rsp=0xc04dded4 rfp=0xc04ddee0
        r7=0x20000124 r6=0x00000004
        r5=0x20000130 r4=0xc0276a40
mtx_pool_setup_dynamic() at mtx_pool_setup_dynamic+0x10
scp=0xc00b2900 rlv=0xc0082654 (mi_startup+0xf8)
        rsp=0xc04ddee4 rfp=0xc04ddef4
mi_startup() at mi_startup+0x10
scp=0xc008256c rlv=0xc00001cc (virt_done+0x18)
        rsp=0xc04ddef8 rfp=0x00000000
        r4=0x2000020c

If you need complete backtraces or other details let me know.

-- Ian