Failed attempt to boot a (non-debug) head -r339076 on an old PowerMac G5 "Quad Core" (built via devel/powerpc64-gcc): Waking up CPU 1

[Nathan Whitehorn]

On 10/9/18 2:07 PM, Andreas Tobler wrote:
> On 09.10.18 22:40, Andreas Tobler wrote:
>> On 09.10.18 22:35, Mark Millard via freebsd-ppc wrote:
>>> [Reverting head -r334498 in my head -r339076 context was enough to get
>>> the G5 so-called "Quad Core" to boot just fine as a variant of
>>> -r339076 .]
>>>
>>> On 2018-Oct-9, at 12:54 PM, Mark Millard <marklmi at yahoo.com> wrote:
>>>
>>>> On 2018-Oct-9, at 8:20 AM, Mark Millard <marklmi at yahoo.com> wrote:
>>>>
>>>>> [The stable/head mix seems to be a wrong idea: 11.2 gets past
>>>>> the SMP: messages just fine on the so-called G5 "Quad Core".]
>>>>>
>>>>> On 2018-Oct-8, at 5:14 PM, Mark Millard <marklmi at yahoo.com> wrote:
>>>>>
>>>>>> On 2018-Oct-8, at 1:27 PM, Justin Hibbits <chmeeedalf at
>>>>>> gmail.com> wrote:
>>>>>>
>>>>>>>> . . .
>>>>>>>
>>>>>>> It would be helpful to know the last known-good SVN revision,
>>>>>>> both for
>>>>>>> Head and 11.x, as well as the oldest failing one.  Since my G5
>>>>>>> bit the
>>>>>>> dust, I can't check locally.
>>>>>>
>>>>> . . .
>>>>
>>>> There are examples of head's kernels that sometimes
>>>> fail to get to the "SMP:" messages and sometimes work
>>>> for getting there (and beyond). So:
>>>>
>>>> My reporting any example failure is a solid indicator
>>>> of the "does not reach "SMP:" problem in that build.
>>>> (All tries reached the waking message on at least cpu
>>>> 1.)
>>>>
>>>> My reporting "worked" for a revision might be a
>>>> misclassification. (This makes for a messier
>>>> "binary-like search".)
>>>>
>>>> That said, the summary of the later detail is:
>>>>
>>>> head -r334494 kernel worked
>>>> head -r334528 kernel failed
>>>>
>>>> (There is nothing between those for:
>>>>
>>>> https://artifact.ci.freebsd.org/snapshot/head/r*/powerpc/powerpc64/kernel.txz
>>>>
>>>>
>>>> so getting a smaller range requires builds.
>>>> I've not attempted that.)
>>>>
>>>> The only machine-dependent powerpc64 change between
>>>> those 2 that I see is:
>>>>
>>>> Author: jhibbits
>>>> Date: Fri Jun  1 21:37:20 2018
>>>> New Revision: 334498
>>>> URL:
>>>> https://svnweb.freebsd.org/changeset/base/334498
>>>>
>>>>
>>>> Log:
>>>>    Increase powerpc64 KVA from ~7.25GB to 32GB
>>>> . . .
>>>>
>>>> . . .
>>>
>>> In my -r339076 build context I reverted -r334498, did a
>>> buildkernel, installed it, and rebooted into -r339076.
>>>
>>> The result booted just fine.
>>>
>>> It does appear that, for head, -r334498 makes the difference
>>> for some reason.
>>
>> Unfortunately I have to confirm your findings.
>
> Mark, how much physical ram do you have? Can you adjust the
> VM_MAX_KERNEL_ADDRESS just below the amount of RAM you have and see if
> -CURRENT boots? Here it does, I have 14GB and I adjusted
> VM_MAX_KERNEL_ADDRESS to 12GB. It is just a trial to find out what is
> happening.
>
> Thanks,
> Andreas

My guess is that the combination of large RAM (so big DMAP) and large
KVA is causing SLB overflows: a combination of these above 16 GB will
make this rate non-zero. These are fine while in the kernel or userspace
-- they get handled transparently -- but could be fatal during a call to
Open Firmware, which, on Apple hardware, operates with the MMU on since
OF doesn't have the appropriate trap handlers to handle DSE/ISE. This
would also explain why the issue only affects Apple hardware. There
would be a few ways to fix it:
1. Distill the OF tree to an FDT before starting the kernel (usefdt=1
option in the loader). There are still issues with this on some Apple
hardware.
2. Right before calling into OF, and after any possibility of  prefault
all the possible SLB entries corresponding to any 32-bit address (all
the ones OF could possibly use) to eliminate the possibility of
in-firmware SLB faults.
3. Leave the kernel SLB fault handlers (+ associated metadata at low
addresses) in place during OF calls.
-Nathan