Cleaned -up evidence about the PowerMac G5 multiprocessor boot hang ups with the modern VM_MAX_KERNEL_ADDRESS value [Wrong radix invaldates prior "new" information]
Mark Millard
marklmi at yahoo.com
Mon Feb 18 03:11:33 UTC 2019
[I used %jx instead of the %jd that I had intended and:
0x2711 == 10001. SO IGNORE MY PRIOR CONCLUSION. In fact
the hang-up has the loop run until timeout==0 .
No evidence of pc_awake being temporarily non-zero.]
On 2019-Feb-17, at 12:10, Mark Millard <marklmi at yahoo.com> wrote:
> [Reporting the how many times the DELAY(100) in
> powermac_smp_start_cpu happens proved very interesting:
> pc_awake for CPU 3 for what turns into a hang does
> temporarily become non-zero as seen on cpu 0 (stopping
> the wait loop) but becomes zero again as seen on CPU 0,
> at least for some hung-up boots.]
>
> On 2019-Feb-16, at 16:29, Mark Millard <marklmi at yahoo.com> wrote:
>
>> I eliminated a bunch of reports that gave information
>> that was the same for hang-ups and successful boots.
>> So this is starting over with less material to go
>> through.
>>
>> For an unsuccessful boot:
>> (Values like 0x10, 0x25, 0x51 are use as recorded labels for points in the code.)
>>
>> Adding CPU 0, hwref=cd38, awake=1
>> Waking up CPU 3 (dev=c480)
>> After reset 4&0 for CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000f0=0x25
>> After attempted wait for awake CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000f0=0x25
>> Waking up CPU 2 (dev=c768)
>>
>> So neither pc_awake nor *0xc0000000000000f0 have machdep_ap_bootstrap
>> based values as seen by CPU 0 (from trying to start CPU 3).
>>
>> For a successful boot:
>>
>> Adding CPU 0, hwref=cd38, awake=1
>> Waking up CPU 3 (dev=c480)
>> After reset 4&0 for CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000f0=0x25
>> After attempted wait for awake CPU 3, hwref=c480, awake=1, *(unsigned long*)0xc0000000000000f0=0x51
>> Adding CPU 3, hwref=c480, awake=1
>> Waking up CPU 2 (dev=c768)
>> After reset 4&0 for CPU 2, hwref=c768, awake=0, *(unsigned long*)0xc0000000000000f0=0x51
>> After attempted wait for awake CPU 2, hwref=c768, awake=1, *(unsigned long*)0xc0000000000000f0=0x51
>> Adding CPU 2, hwref=c768, awake=1
>> Waking up CPU 1 (dev=ca50)
>> After reset 4&0 for CPU 1, hwref=ca50, awake=0, *(unsigned long*)0xc0000000000000f0=0x51
>> After attempted wait for awake CPU 1, hwref=ca50, awake=1, *(unsigned long*)0xc0000000000000f0=0x51
>> Adding CPU 1, hwref=ca50, awake=1
>> SMP: AP CPU #3 launched
>> SMP: AP CPU #1 launched
>> SMP: AP CPU #2 launched
>> Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
>> GEOM: new disk ada1
>>
>> So here waiting on cpu 0 does see pc_awake become 1 and does see
>> 0xc0000000000000f0 updated by machdep_ap_bootstrap for CPU , for
>> example.
>>
>> Note the awake=0 (i.e.., pc_awake) examples for each cpu even when
>> there is already a 0x51 label for CPU 2 and CPU 3. (The code is
>> shown later.)
>>
>> This is based on the labeling updates to:
>> ("labels" stored at 0xc0000000000000f0)
>>
>> cpudep_ap_early_bootstrap (before-return labeled with value : 0x10)
>> moea64_cpu_bootstrap_native (labeled with values: 0x21, 0x22, 0x23, 0x24, 0x25 at various points)
>> pmap_cpu_bootstrap (before-return labeled with value : 0x20)
>> cpudep_ap_bootstrap (before-return labeled with value : 0x30)
>> cpudep_ap_setup (before-return labeled with value : 0x40)
>> machdep_ap_bootstrap (lableled with values: 0x5F, 0x51, and 0x50 at various points)
>>
>> powermac_smp_start_cpu (reporting values after the resets for a CPU and after
>> the pa_awake loop)
>> . . .
>>
>
> From a successful boot:
> (note the "delay 100 count = 0," exmaples)
>
> Adding CPU 0, hwref=cd38, awake=1
> Waking up CPU 3 (dev=c480)
> After reset 4&0 for CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000f0=0x25
> After attempted wait for awake CPU 3, hwref=c480, awake=1, delay 100 count = 0, *(unsigned long*)0xc0000000000000f0=0x51
> Adding CPU 3, hwref=c480, awake=1
> Waking up CPU 2 (dev=c768)
> After reset 4&0 for CPU 2, hwref=c768, awake=0, *(unsigned long*)0xc0000000000000f0=0x51
> After attempted wait for awake CPU 2, hwref=c768, awake=1, delay 100 count = 0, *(unsigned long*)0xc0000000000000f0=0x51
> Adding CPU 2, hwref=c768, awake=1
> Waking up CPU 1 (dev=ca50)
> After reset 4&0 for CPU 1, hwref=ca50, awake=0, *(unsigned long*)0xc0000000000000f0=0x51
> After attempted wait for awake CPU 1, hwref=ca50, awake=1, delay 100 count = 0, *(unsigned long*)0xc0000000000000f0=0x51
> Adding CPU 1, hwref=ca50, awake=1
> SMP: AP CPU #1 launched
> SMP: AP CPU #2 launched
> SMP: AP CPU #3 launched
> Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
> GEOM: new disk ada1
>
>
> Examples of boot hung ups look something like:
> (note the "delay 100 count = 2711," exmaples)
>
> Adding CPU 0, hwref=cd38, awake=1
> Waking up CPU 3 (dev=c480)
> After reset 4&0 for CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000f0=0x25
> After attempted wait for awake CPU 3, hwref=c480, awake=0, delay 100 count = 2711, *(unsigned long*)0xc0000000000000f0=0x25
> Waking up CPU 2 (dev=c768)
>
> (The above is from a picture of the screen.)
The following is wrong because 2711 was hexadecimal,
instead of the decimal I had intended:
> 2711 implies that pc_awake was for a time non-zero. The
> code in question is:
0x2711 == 10001, indicating that the loop ran to its
conclusion via timeout==0 .
> if (bootverbose) // HACK!!!
> printf("After reset 4&0 for CPU %d, hwref=%jx, awake=%x, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
> pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
> pc->pc_awake,(uintmax_t)*(unsigned long*)0xc0000000000000f0);
>
> timeout = 10000;
> while (!pc->pc_awake && timeout--)
> DELAY(100);
>
> if (bootverbose) // HACK!!!
> printf("After attempted wait for awake CPU %d, hwref=%jx, awake=%x, delay 100 count = %jx, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
> pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
> pc->pc_awake, (uintmax_t)(10000-timeout), (uintmax_t)*(unsigned long*)0xc0000000000000f0);
Above: I used %jx for the count instead of the intended %jd.
> (If the loop stops for timeout==0 the calculation of the count will be
> high by 1.)
>
>
>
> Side note:
>
> I've also eliminated the experimental, extra isync() additions.
> The tested code is below.
>
> Index: /usr/src/sys/powerpc/aim/moea64_native.c
> ===================================================================
> --- /usr/src/sys/powerpc/aim/moea64_native.c (revision 344018)
> +++ /usr/src/sys/powerpc/aim/moea64_native.c (working copy)
> @@ -384,6 +384,9 @@
>
> mtmsr(mfmsr() & ~PSL_DR & ~PSL_IR);
>
> + *(unsigned long*)0xc0000000000000f0 = 0x21; // HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> /*
> * Install kernel SLB entries
> */
> @@ -393,6 +396,9 @@
> __asm __volatile ("slbmfee %0,%1; slbie %0;" : "=r"(seg0) :
> "r"(0));
>
> + *(unsigned long*)0xc0000000000000f0 = 0x22; // HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> for (i = 0; i < n_slbs; i++) {
> if (!(slb[i].slbe & SLBE_VALID))
> continue;
> @@ -400,6 +406,9 @@
> __asm __volatile ("slbmte %0, %1" ::
> "r"(slb[i].slbv), "r"(slb[i].slbe));
> }
> +
> + *(unsigned long*)0xc0000000000000f0 = 0x23; // HACK!!!
> + powerpc_sync(); // HACK!!!
> #else
> for (i = 0; i < 16; i++)
> mtsrin(i << ADDR_SR_SHFT, kernel_pmap->pm_sr[i]);
> @@ -412,7 +421,14 @@
> __asm __volatile ("ptesync; mtsdr1 %0; isync"
> :: "r"(((uintptr_t)moea64_pteg_table & ~DMAP_BASE_ADDRESS)
> | (uintptr_t)(flsl(moea64_pteg_mask >> 11))));
> +
> + *(unsigned long*)0xc0000000000000f0 = 0x24; // HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> tlbia();
> +
> + *(unsigned long*)0xc0000000000000f0 = 0x25; // HACK!!!
> + powerpc_sync(); // HACK!!!
> }
>
> static void
> Index: /usr/src/sys/powerpc/aim/mp_cpudep.c
> ===================================================================
> --- /usr/src/sys/powerpc/aim/mp_cpudep.c (revision 344018)
> +++ /usr/src/sys/powerpc/aim/mp_cpudep.c (working copy)
> @@ -105,6 +105,9 @@
>
> __asm __volatile("mtsprg 0, %0" :: "r"(ap_pcpu));
> powerpc_sync();
> +
> + *(unsigned long*)0xc0000000000000f0 = 0x10; // HACK!!!
> + powerpc_sync(); // HACK!!!
> }
>
> uintptr_t
> @@ -124,6 +127,9 @@
> pcpup->pc_curpcb = pcpup->pc_curthread->td_pcb;
> sp = pcpup->pc_curpcb->pcb_sp;
>
> + *(unsigned long*)0xc0000000000000f0 = 0x30; // HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> return (sp);
> }
>
> @@ -416,5 +422,8 @@
> "suboptimal.\n");
> break;
> }
> +
> + *(unsigned long*)0xc0000000000000f0 = 0x40; // HACK!!!
> + powerpc_sync(); // HACK!!!
> }
>
> Index: /usr/src/sys/powerpc/include/vmparam.h
> ===================================================================
> --- /usr/src/sys/powerpc/include/vmparam.h (revision 344018)
> +++ /usr/src/sys/powerpc/include/vmparam.h (working copy)
> @@ -109,9 +109,11 @@
> #ifndef LOCORE
> #define VM_MIN_KERNEL_ADDRESS 0xe000000000000000UL
> #define VM_MAX_KERNEL_ADDRESS 0xe0000007ffffffffUL
> +//#define VM_MAX_KERNEL_ADDRESS 0xe0000001c7ffffffUL
> #else
> #define VM_MIN_KERNEL_ADDRESS 0xe000000000000000
> #define VM_MAX_KERNEL_ADDRESS 0xe0000007ffffffff
> +//#define VM_MAX_KERNEL_ADDRESS 0xe0000001c7ffffff
> #endif
> #define VM_MAX_SAFE_KERNEL_ADDRESS VM_MAX_KERNEL_ADDRESS
> #endif
> Index: /usr/src/sys/powerpc/powermac/platform_powermac.c
> ===================================================================
> --- /usr/src/sys/powerpc/powermac/platform_powermac.c (revision 344018)
> +++ /usr/src/sys/powerpc/powermac/platform_powermac.c (working copy)
> @@ -382,10 +382,20 @@
> (void)(*rstvec);
> powerpc_sync();
>
> + if (bootverbose) // HACK!!!
> + printf("After reset 4&0 for CPU %d, hwref=%jx, awake=%x, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
> + pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
> + pc->pc_awake,(uintmax_t)*(unsigned long*)0xc0000000000000f0);
> +
> timeout = 10000;
> while (!pc->pc_awake && timeout--)
> DELAY(100);
>
> + if (bootverbose) // HACK!!!
> + printf("After attempted wait for awake CPU %d, hwref=%jx, awake=%x, delay 100 count = %jx, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
> + pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
> + pc->pc_awake, (uintmax_t)(10000-timeout), (uintmax_t)*(unsigned long*)0xc0000000000000f0);
> +
> return ((pc->pc_awake) ? 0 : EBUSY);
> #else
> /* No SMP support */
Again (above): I used %jx for the count instead of the intended
%jd .
> Index: /usr/src/sys/powerpc/powerpc/machdep.c
> ===================================================================
> --- /usr/src/sys/powerpc/powerpc/machdep.c (revision 344018)
> +++ /usr/src/sys/powerpc/powerpc/machdep.c (working copy)
> @@ -141,8 +141,6 @@
> extern vm_paddr_t kernload;
> #endif
>
> -extern void *ap_pcpu;
> -
> struct pcpu __pcpu[MAXCPU];
> static char init_kenv[2048];
>
> Index: /usr/src/sys/powerpc/powerpc/mp_machdep.c
> ===================================================================
> --- /usr/src/sys/powerpc/powerpc/mp_machdep.c (revision 344018)
> +++ /usr/src/sys/powerpc/powerpc/mp_machdep.c (working copy)
> @@ -73,10 +73,15 @@
> void
> machdep_ap_bootstrap(void)
> {
> + *(unsigned long*)0xc0000000000000f0 = 0x5F; // HACK!!!
> + powerpc_sync(); // HACK!!!
>
> PCPU_SET(awake, 1);
> __asm __volatile("msync; isync");
>
> + *(unsigned long*)0xc0000000000000f0 = 0x51; // HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> while (ap_letgo == 0)
> __asm __volatile("or 31,31,31");
> __asm __volatile("or 6,6,6");
> @@ -113,6 +118,9 @@
> /* Start per-CPU event timers. */
> cpu_initclocks_ap();
>
> + *(unsigned long*)0xc0000000000000f0 = 0x50; // HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> /* Announce ourselves awake, and enter the scheduler */
> sched_throw(NULL);
> }
> @@ -256,6 +264,7 @@
> }
>
> ap_awake = 1;
> + powerpc_sync(); // HACK!!!
>
> /* Provide our current DEC and TB values for APs */
> ap_timebase = mftb() + 10;
> Index: /usr/src/sys/powerpc/powerpc/pmap_dispatch.c
> ===================================================================
> --- /usr/src/sys/powerpc/powerpc/pmap_dispatch.c (revision 344018)
> +++ /usr/src/sys/powerpc/powerpc/pmap_dispatch.c (working copy)
> @@ -445,6 +445,9 @@
> */
>
> return (MMU_CPU_BOOTSTRAP(mmu_obj, ap));
> +
> + *(unsigned long*)0xc0000000000000f0 = 0x20; // HACK!!!
> + powerpc_sync(); // HACK!!!
> }
>
> void *
>
>
>
===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
More information about the freebsd-ppc
mailing list