Cleaned -up evidence about the PowerMac G5 multiprocessor boot hang ups with the modern VM_MAX_KERNEL_ADDRESS value [signficant new evidence]
Mark Millard
marklmi at yahoo.com
Sun Feb 17 20:10:25 UTC 2019
[Reporting the how many times the DELAY(100) in
powermac_smp_start_cpu happens proved very interesting:
pc_awake for CPU 3 for what turns into a hang does
temporarily become non-zero as seen on cpu 0 (stopping
the wait loop) but becomes zero again as seen on CPU 0,
at least for some hung-up boots.]
On 2019-Feb-16, at 16:29, Mark Millard <marklmi at yahoo.com> wrote:
> I eliminated a bunch of reports that gave information
> that was the same for hang-ups and successful boots.
> So this is starting over with less material to go
> through.
>
> For an unsuccessful boot:
> (Values like 0x10, 0x25, 0x51 are use as recorded labels for points in the code.)
>
> Adding CPU 0, hwref=cd38, awake=1
> Waking up CPU 3 (dev=c480)
> After reset 4&0 for CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000f0=0x25
> After attempted wait for awake CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000f0=0x25
> Waking up CPU 2 (dev=c768)
>
> So neither pc_awake nor *0xc0000000000000f0 have machdep_ap_bootstrap
> based values as seen by CPU 0 (from trying to start CPU 3).
>
> For a successful boot:
>
> Adding CPU 0, hwref=cd38, awake=1
> Waking up CPU 3 (dev=c480)
> After reset 4&0 for CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000f0=0x25
> After attempted wait for awake CPU 3, hwref=c480, awake=1, *(unsigned long*)0xc0000000000000f0=0x51
> Adding CPU 3, hwref=c480, awake=1
> Waking up CPU 2 (dev=c768)
> After reset 4&0 for CPU 2, hwref=c768, awake=0, *(unsigned long*)0xc0000000000000f0=0x51
> After attempted wait for awake CPU 2, hwref=c768, awake=1, *(unsigned long*)0xc0000000000000f0=0x51
> Adding CPU 2, hwref=c768, awake=1
> Waking up CPU 1 (dev=ca50)
> After reset 4&0 for CPU 1, hwref=ca50, awake=0, *(unsigned long*)0xc0000000000000f0=0x51
> After attempted wait for awake CPU 1, hwref=ca50, awake=1, *(unsigned long*)0xc0000000000000f0=0x51
> Adding CPU 1, hwref=ca50, awake=1
> SMP: AP CPU #3 launched
> SMP: AP CPU #1 launched
> SMP: AP CPU #2 launched
> Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
> GEOM: new disk ada1
>
> So here waiting on cpu 0 does see pc_awake become 1 and does see
> 0xc0000000000000f0 updated by machdep_ap_bootstrap for CPU , for
> example.
>
> Note the awake=0 (i.e.., pc_awake) examples for each cpu even when
> there is already a 0x51 label for CPU 2 and CPU 3. (The code is
> shown later.)
>
> This is based on the labeling updates to:
> ("labels" stored at 0xc0000000000000f0)
>
> cpudep_ap_early_bootstrap (before-return labeled with value : 0x10)
> moea64_cpu_bootstrap_native (labeled with values: 0x21, 0x22, 0x23, 0x24, 0x25 at various points)
> pmap_cpu_bootstrap (before-return labeled with value : 0x20)
> cpudep_ap_bootstrap (before-return labeled with value : 0x30)
> cpudep_ap_setup (before-return labeled with value : 0x40)
> machdep_ap_bootstrap (lableled with values: 0x5F, 0x51, and 0x50 at various points)
>
> powermac_smp_start_cpu (reporting values after the resets for a CPU and after
> the pa_awake loop)
> . . .
>
From a successful boot:
(note the "delay 100 count = 0," exmaples)
Adding CPU 0, hwref=cd38, awake=1
Waking up CPU 3 (dev=c480)
After reset 4&0 for CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000f0=0x25
After attempted wait for awake CPU 3, hwref=c480, awake=1, delay 100 count = 0, *(unsigned long*)0xc0000000000000f0=0x51
Adding CPU 3, hwref=c480, awake=1
Waking up CPU 2 (dev=c768)
After reset 4&0 for CPU 2, hwref=c768, awake=0, *(unsigned long*)0xc0000000000000f0=0x51
After attempted wait for awake CPU 2, hwref=c768, awake=1, delay 100 count = 0, *(unsigned long*)0xc0000000000000f0=0x51
Adding CPU 2, hwref=c768, awake=1
Waking up CPU 1 (dev=ca50)
After reset 4&0 for CPU 1, hwref=ca50, awake=0, *(unsigned long*)0xc0000000000000f0=0x51
After attempted wait for awake CPU 1, hwref=ca50, awake=1, delay 100 count = 0, *(unsigned long*)0xc0000000000000f0=0x51
Adding CPU 1, hwref=ca50, awake=1
SMP: AP CPU #1 launched
SMP: AP CPU #2 launched
SMP: AP CPU #3 launched
Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
GEOM: new disk ada1
Examples of boot hung ups look something like:
(note the "delay 100 count = 2711," exmaples)
Adding CPU 0, hwref=cd38, awake=1
Waking up CPU 3 (dev=c480)
After reset 4&0 for CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000f0=0x25
After attempted wait for awake CPU 3, hwref=c480, awake=0, delay 100 count = 2711, *(unsigned long*)0xc0000000000000f0=0x25
Waking up CPU 2 (dev=c768)
(The above is from a picture of the screen.)
2711 implies that pc_awake was for a time non-zero. The
code in question is:
if (bootverbose) // HACK!!!
printf("After reset 4&0 for CPU %d, hwref=%jx, awake=%x, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
pc->pc_awake,(uintmax_t)*(unsigned long*)0xc0000000000000f0);
timeout = 10000;
while (!pc->pc_awake && timeout--)
DELAY(100);
if (bootverbose) // HACK!!!
printf("After attempted wait for awake CPU %d, hwref=%jx, awake=%x, delay 100 count = %jx, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
pc->pc_awake, (uintmax_t)(10000-timeout), (uintmax_t)*(unsigned long*)0xc0000000000000f0);
(If the loop stops for timeout==0 the calculation of the count will be
high by 1.)
Side note:
I've also eliminated the experimental, extra isync() additions.
The tested code is below.
Index: /usr/src/sys/powerpc/aim/moea64_native.c
===================================================================
--- /usr/src/sys/powerpc/aim/moea64_native.c (revision 344018)
+++ /usr/src/sys/powerpc/aim/moea64_native.c (working copy)
@@ -384,6 +384,9 @@
mtmsr(mfmsr() & ~PSL_DR & ~PSL_IR);
+ *(unsigned long*)0xc0000000000000f0 = 0x21; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
/*
* Install kernel SLB entries
*/
@@ -393,6 +396,9 @@
__asm __volatile ("slbmfee %0,%1; slbie %0;" : "=r"(seg0) :
"r"(0));
+ *(unsigned long*)0xc0000000000000f0 = 0x22; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
for (i = 0; i < n_slbs; i++) {
if (!(slb[i].slbe & SLBE_VALID))
continue;
@@ -400,6 +406,9 @@
__asm __volatile ("slbmte %0, %1" ::
"r"(slb[i].slbv), "r"(slb[i].slbe));
}
+
+ *(unsigned long*)0xc0000000000000f0 = 0x23; // HACK!!!
+ powerpc_sync(); // HACK!!!
#else
for (i = 0; i < 16; i++)
mtsrin(i << ADDR_SR_SHFT, kernel_pmap->pm_sr[i]);
@@ -412,7 +421,14 @@
__asm __volatile ("ptesync; mtsdr1 %0; isync"
:: "r"(((uintptr_t)moea64_pteg_table & ~DMAP_BASE_ADDRESS)
| (uintptr_t)(flsl(moea64_pteg_mask >> 11))));
+
+ *(unsigned long*)0xc0000000000000f0 = 0x24; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
tlbia();
+
+ *(unsigned long*)0xc0000000000000f0 = 0x25; // HACK!!!
+ powerpc_sync(); // HACK!!!
}
static void
Index: /usr/src/sys/powerpc/aim/mp_cpudep.c
===================================================================
--- /usr/src/sys/powerpc/aim/mp_cpudep.c (revision 344018)
+++ /usr/src/sys/powerpc/aim/mp_cpudep.c (working copy)
@@ -105,6 +105,9 @@
__asm __volatile("mtsprg 0, %0" :: "r"(ap_pcpu));
powerpc_sync();
+
+ *(unsigned long*)0xc0000000000000f0 = 0x10; // HACK!!!
+ powerpc_sync(); // HACK!!!
}
uintptr_t
@@ -124,6 +127,9 @@
pcpup->pc_curpcb = pcpup->pc_curthread->td_pcb;
sp = pcpup->pc_curpcb->pcb_sp;
+ *(unsigned long*)0xc0000000000000f0 = 0x30; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
return (sp);
}
@@ -416,5 +422,8 @@
"suboptimal.\n");
break;
}
+
+ *(unsigned long*)0xc0000000000000f0 = 0x40; // HACK!!!
+ powerpc_sync(); // HACK!!!
}
Index: /usr/src/sys/powerpc/include/vmparam.h
===================================================================
--- /usr/src/sys/powerpc/include/vmparam.h (revision 344018)
+++ /usr/src/sys/powerpc/include/vmparam.h (working copy)
@@ -109,9 +109,11 @@
#ifndef LOCORE
#define VM_MIN_KERNEL_ADDRESS 0xe000000000000000UL
#define VM_MAX_KERNEL_ADDRESS 0xe0000007ffffffffUL
+//#define VM_MAX_KERNEL_ADDRESS 0xe0000001c7ffffffUL
#else
#define VM_MIN_KERNEL_ADDRESS 0xe000000000000000
#define VM_MAX_KERNEL_ADDRESS 0xe0000007ffffffff
+//#define VM_MAX_KERNEL_ADDRESS 0xe0000001c7ffffff
#endif
#define VM_MAX_SAFE_KERNEL_ADDRESS VM_MAX_KERNEL_ADDRESS
#endif
Index: /usr/src/sys/powerpc/powermac/platform_powermac.c
===================================================================
--- /usr/src/sys/powerpc/powermac/platform_powermac.c (revision 344018)
+++ /usr/src/sys/powerpc/powermac/platform_powermac.c (working copy)
@@ -382,10 +382,20 @@
(void)(*rstvec);
powerpc_sync();
+ if (bootverbose) // HACK!!!
+ printf("After reset 4&0 for CPU %d, hwref=%jx, awake=%x, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
+ pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
+ pc->pc_awake,(uintmax_t)*(unsigned long*)0xc0000000000000f0);
+
timeout = 10000;
while (!pc->pc_awake && timeout--)
DELAY(100);
+ if (bootverbose) // HACK!!!
+ printf("After attempted wait for awake CPU %d, hwref=%jx, awake=%x, delay 100 count = %jx, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
+ pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
+ pc->pc_awake, (uintmax_t)(10000-timeout), (uintmax_t)*(unsigned long*)0xc0000000000000f0);
+
return ((pc->pc_awake) ? 0 : EBUSY);
#else
/* No SMP support */
Index: /usr/src/sys/powerpc/powerpc/machdep.c
===================================================================
--- /usr/src/sys/powerpc/powerpc/machdep.c (revision 344018)
+++ /usr/src/sys/powerpc/powerpc/machdep.c (working copy)
@@ -141,8 +141,6 @@
extern vm_paddr_t kernload;
#endif
-extern void *ap_pcpu;
-
struct pcpu __pcpu[MAXCPU];
static char init_kenv[2048];
Index: /usr/src/sys/powerpc/powerpc/mp_machdep.c
===================================================================
--- /usr/src/sys/powerpc/powerpc/mp_machdep.c (revision 344018)
+++ /usr/src/sys/powerpc/powerpc/mp_machdep.c (working copy)
@@ -73,10 +73,15 @@
void
machdep_ap_bootstrap(void)
{
+ *(unsigned long*)0xc0000000000000f0 = 0x5F; // HACK!!!
+ powerpc_sync(); // HACK!!!
PCPU_SET(awake, 1);
__asm __volatile("msync; isync");
+ *(unsigned long*)0xc0000000000000f0 = 0x51; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
while (ap_letgo == 0)
__asm __volatile("or 31,31,31");
__asm __volatile("or 6,6,6");
@@ -113,6 +118,9 @@
/* Start per-CPU event timers. */
cpu_initclocks_ap();
+ *(unsigned long*)0xc0000000000000f0 = 0x50; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
/* Announce ourselves awake, and enter the scheduler */
sched_throw(NULL);
}
@@ -256,6 +264,7 @@
}
ap_awake = 1;
+ powerpc_sync(); // HACK!!!
/* Provide our current DEC and TB values for APs */
ap_timebase = mftb() + 10;
Index: /usr/src/sys/powerpc/powerpc/pmap_dispatch.c
===================================================================
--- /usr/src/sys/powerpc/powerpc/pmap_dispatch.c (revision 344018)
+++ /usr/src/sys/powerpc/powerpc/pmap_dispatch.c (working copy)
@@ -445,6 +445,9 @@
*/
return (MMU_CPU_BOOTSTRAP(mmu_obj, ap));
+
+ *(unsigned long*)0xc0000000000000f0 = 0x20; // HACK!!!
+ powerpc_sync(); // HACK!!!
}
void *
===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
More information about the freebsd-ppc
mailing list