Some evidence about the PowerMac G5 multiprocessor boot hang ups with the modern VM_MAX_KERNEL_ADDRESS value [found more staging info]
Mark Millard
marklmi at yahoo.com
Sat Feb 16 20:07:29 UTC 2019
[I needed to allow more time after the 2 resets before
having CPU 0 look at the memory. It was reporting
older values instead of my added writes. The odd
non-zero value was from before the activity of interest.]
I start with the new result found, then give supporting
material.
I've now seen hangs with:
*(unsigned long*)0xc0000000000000f0)=0x10
for CPU 3. So the following completed:
void
cpudep_ap_early_bootstrap(void)
{
#ifndef __powerpc64__
register_t reg;
#endif
switch (mfpvr() >> 16) {
case IBM970:
case IBM970FX:
case IBM970MP:
/* Restore HID4 and HID5, which are necessary for the MMU */
#ifdef __powerpc64__
mtspr(SPR_HID4, bsp_state[2]); powerpc_sync(); isync();
mtspr(SPR_HID5, bsp_state[3]); powerpc_sync(); isync();
#else
__asm __volatile("ld %0, 16(%2); sync; isync; \
mtspr %1, %0; sync; isync;"
: "=r"(reg) : "K"(SPR_HID4), "b"(bsp_state));
__asm __volatile("ld %0, 24(%2); sync; isync; \
mtspr %1, %0; sync; isync;"
: "=r"(reg) : "K"(SPR_HID5), "b"(bsp_state));
#endif
powerpc_sync();
break;
case IBMPOWER8:
case IBMPOWER8E:
case IBMPOWER9:
#ifdef __powerpc64__
if (mfmsr() & PSL_HV) {
isync();
/*
* Direct interrupts to SRR instead of HSRR and
* reset LPCR otherwise
*/
mtspr(SPR_LPID, 0);
isync();
mtspr(SPR_LPCR, lpcr);
isync();
}
#endif
break;
}
__asm __volatile("mtsprg 0, %0" :: "r"(ap_pcpu));
powerpc_sync();
*(unsigned long*)0xc0000000000000f0 = 0x10; // HACK!!!
powerpc_sync(); // HACK!!!
}
but the following (and later) did not complete:
void
pmap_cpu_bootstrap(int ap)
{
/*
* No KTR here because our console probably doesn't work yet
*/
return (MMU_CPU_BOOTSTRAP(mmu_obj, ap));
*(unsigned long*)0xc0000000000000f0 = 0x20; // HACK!!!
powerpc_sync(); // HACK!!!
}
Background for reference relative to showing
the intended values. . .
I now have:
*rstvec = 4;
powerpc_sync();
(void)(*rstvec);
powerpc_sync();
DELAY(1);
*rstvec = 0;
powerpc_sync();
(void)(*rstvec);
powerpc_sync();
if (bootverbose) // HACK!!!
printf("After reset 4&0 for CPU %d, hwref=%jx, awake=%x, *(unsigned long*)0xc0000000000000e0=0x%jx, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
pc->pc_awake,(uintmax_t)*(unsigned long*)0xc0000000000000e0,(uintmax_t)*(unsigned long*)0xc0000000000000f0);
timeout = 10000;
while (!pc->pc_awake && timeout--)
DELAY(100);
if (bootverbose) // HACK!!!
printf("After attempted wait for awake CPU %d, hwref=%jx, awake=%x, *(unsigned long*)0xc0000000000000e0=0x%jx, *(unsigned long*)0xc0000000000000f0=0x%jx\n",
pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
pc->pc_awake,(uintmax_t)*(unsigned long*)0xc0000000000000e0,(uintmax_t)*(unsigned long*)0xc0000000000000f0);
The 2nd printf shows the expected values but the first
above shows old memory values.
The memory at 0xc0000000000000e0 hold the get_pcpu() result.
The memory at 0xc0000000000000f0 hold the 0x?? values that I'd reported earlier
although I've added a 0x5E but really use the value at 0xc0000000000000e0 to
check since 0x5e would be replaced by 0x51:
struct pcpu* thepcpu = get_pcpu(); // HACK!!!
*(struct pcpu**)0xc0000000000000e0 = thepcpu; // HACK!!!
if ( thepcpu==&__pcpu[1] // HACK!!!
|| thepcpu==&__pcpu[2]
|| thepcpu==&__pcpu[3]
)
*(unsigned long*)0xc0000000000000f0 = 0x5F; // HACK!!!
else
*(unsigned long*)0xc0000000000000f0 = 0x5E; // HACK!!!
powerpc_sync(); // HACK!!!
PCPU_SET(awake, 1);
__asm __volatile("msync; isync");
*(unsigned long*)0xc0000000000000f0 = 0x51; // HACK!!!
powerpc_sync(); // HACK!!!
while (ap_letgo == 0)
__asm __volatile("or 31,31,31");
__asm __volatile("or 6,6,6");
For reference, a successful boot now looks like:
Adding CPU 0, hwref=cd38, awake=1
Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
Waking up CPU 3 (dev=c480)
powermac_smp_start_cpu 's OF_getprop for CPU 3, hwref=c480, awake=0: res=4, reset=8c
powermac_smp_start_cpu for CPU 3, hwref=c480, awake=0: rstvec_virtbase=0xe000000087fd2000
powermac_smp_start_cpu for CPU 3, hwref=c480, awake=0: rstvec=0xe000000087fd208c
Before reset 4&0 for CPU 3, hwref=c480, awake=0
After reset 4&0 for CPU 3, hwref=c480, awake=0, *(unsigned long*)0xc0000000000000e0=0x0, *(unsigned long*)0xc0000000000000f0=0x0
After attempted wait for awake CPU 3, hwref=c480, awake=1, *(unsigned long*)0xc0000000000000e0=0xc0000000016c6100, *(unsigned long*)0xc0000000000000f0=0x51
cpu_mp_unleash attempting to wait for pc_awake: CPU 3, hwref=c480, awake=1
cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 3, hwref=c480, awake=1
Adding CPU 3, hwref=c480, awake=1
Waking up CPU 2 (dev=c768)
powermac_smp_start_cpu 's OF_getprop for CPU 2, hwref=c768, awake=0: res=4, reset=8b
powermac_smp_start_cpu for CPU 2, hwref=c768, awake=0: rstvec=0xe000000087fd208b
Before reset 4&0 for CPU 2, hwref=c768, awake=0
After reset 4&0 for CPU 2, hwref=c768, awake=0, *(unsigned long*)0xc0000000000000e0=0xc0000000016c6100, *(unsigned long*)0xc0000000000000f0=0x51
After attempted wait for awake CPU 2, hwref=c768, awake=1, *(unsigned long*)0xc0000000000000e0=0xc0000000016c5100, *(unsigned long*)0xc0000000000000f0=0x51
cpu_mp_unleash attempting to wait for pc_awake: CPU 2, hwref=c768, awake=1
cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 2, hwref=c768, awake=1
Adding CPU 2, hwref=c768, awake=1
Waking up CPU 1 (dev=ca50)
powermac_smp_start_cpu 's OF_getprop for CPU 1, hwref=ca50, awake=0: res=4, reset=8a
powermac_smp_start_cpu for CPU 1, hwref=ca50, awake=0: rstvec=0xe000000087fd208a
Before reset 4&0 for CPU 1, hwref=ca50, awake=0
After reset 4&0 for CPU 1, hwref=ca50, awake=0, *(unsigned long*)0xc0000000000000e0=0xc0000000016c5100, *(unsigned long*)0xc0000000000000f0=0x51
After attempted wait for awake CPU 1, hwref=ca50, awake=1, *(unsigned long*)0xc0000000000000e0=0xc0000000016c4100, *(unsigned long*)0xc0000000000000f0=0x51
cpu_mp_unleash attempting to wait for pc_awake: CPU 1, hwref=ca50, awake=1
cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 1, hwref=ca50, awake=1
Adding CPU 1, hwref=ca50, awake=1
machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #2 launched
machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #1 launched
machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #3 launched
SMP: AP CPU #2 launched
SMP: AP CPU #1 launched
SMP: AP CPU #3 launched
machdep_ap_bootstrap after smp_started!=0: AP CPU #2 launched
machdep_ap_bootstrap after smp_started!=0: AP CPU #1 launched
machdep_ap_bootstrap after smp_started!=0: AP CPU #3 launched
===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
More information about the freebsd-ppc
mailing list