TARGET_ARCH=powerpc64 jump from head -r326192 to -r327075: g_event crashes with "instruction storage interrupt" [clang-built kernel specific problem]

[Mark Millard]

[Using devel/powerpc64-gcc to build the kernel
produces a 64-bit kernel that does not crash for
kldload use. clang 5.0.1 may be producing code
that needs a "new" form of relocation compared
to what kldload activity historically needed.]

On 2017-Dec-23, at 9:42 PM, Mark Millard <markmi at dsl-only.net> wrote:

> [TARGET_ARCH=powerpc (gcc 4.2.1 based) does
> not have the problem. And my notes about
> geom_label being builtin were wrong about
> boot usage of labels when geom_label_load
> is not used /boot/loader.conf .]
> 
> On 2017-Dec-23, at 8:00 PM, Mark Millard <markmi at dsl-only.net> wrote:
> 
>> [I note some odd consequences when trying to
>> build in geom_label to the kernel. filemon
>> being built-in work fine for my use.]
>> 
>> On 2017-Dec-22, at 10:33 PM, Mark Millard <markmi at dsl-only.net> wrote:
>> 
>>> [I attempted something that involved kldload -n filemon .
>>> That gives a similar system crash (by the virtual address
>>> reported and the type of exception). It seems similar to
>>> geom_label_load="YES" in /boot/loader.conf getting a
>>> crash.]
>>> 
>>> On 2017-Dec-22, at 9:58 PM, Mark Millard <markmi at dsl-only.net> wrote:
>>> 
>>>> [Avoiding referencing partitions via labels
>>>> avoided the crash.]
>>>> 
>>>> On 2017-Dec-22, at 9:26 PM, Mark Millard <markmi at dsl-only.net> wrote:
>>>> 
>>>>> [Note: I experiment with using clang as the build and
>>>>> system compiler for TARGET_ARCH=powerpc64 .]
>>>>> 
>>>>> I attempted to update a old so-called PowerMac "Quad Core"
>>>>> from head -r326192 to -r328075, noting special instructions
>>>>> that have been published. (This was a non-debug kernel
>>>>> build.)
>>>>> 
>>>>> Unfortunately around the:
>>>>> 
>>>>> . . .
>>>>> cd0: 66.700MB/s transfers (UDMA4, ATPI 12bytes, PIO 65534bytes)
>>>>> cd0: Attempt
>>>>> Trying to mount from ufs:/dev/ufs/FBSDG5Lrootfs [rw,noatime]. . .
>>>>> 
>>>>> There ends up being a repeatable kernel trap:
>>>>> (transcribed from pictures, but with notes added)
>>>>> 
>>>>> fatal kernel trap:
>>>>> (NOTE: The above can be the line before the "Trying to mount" line,
>>>>> after the "cd0: Attempt" line.)
>>>>> 
>>>>> exception       = 0x400 (instruction storage interrupt)
>>>>> virtual address = 0x3c4c009438427518
>>>>> srr0            = 0x3c4c009438427518
>>>>> srr1            = 0x9000000040009032
>>>>> lr              = 0x14234e8
>>>>> curthread       = 0x3d52a80
>>>>> pid = 13, comm = g_event
>>>>> 
>>>>> [ thread pid 13 tid 100019 ]
>>>>> Stopped at 0x3c4c009438427518
>>>>> 
>>>>> It reports:
>>>>> 
>>>>> Tracing command geom pid 13 tid 100019 td 0x3d52a80 (CPU 0)
>>>>> 0xC0000000032966b0: at g_new_provider_event+0x144
>>>>> 0xC0000000032966f0: at g_run_events+0x530
>>>>> 0xC000000003296830: at g_event_procbody+0x94
>>>>> 0xC000000003296860: at fork_exit+0xd8
>>>>> 0xC0000000032968f0: at fork_trampoline+0x18
>>>>> 0xC000000003296920: at blocked_loop+0x38
>>>>> 
>>>>> These details do not vary between attempts.
>>>>> 
>>>>> From what I gather the code jumped to 0x3c4c009438427518
>>>>> but that is not from an executable page for the context
>>>>> at the time.
>>>>> 
>>>>> On the -r326192 powerpc system /dev/ufs/FBSDG5Lrootfs
>>>>> mounts just fine.
>>>>> 
>>>> 
>>>> I adjusted:
>>>> 
>>>> /boot/loader.conf
>>>> /etc/rc.conf
>>>> 
>>>> /etc/fstab
>>>> 
>>>> to avoid use of:
>>>> 
>>>> geom_label_load="YES"
>>>> dumpdev="/dev/label/FBSDG5Lswap"
>>>> 
>>>> /dev/ufs/FBSDG5Lrootfs /               ufs     rw,noatime      1       1
>>>> /dev/label/FBSDG5Lswap none            swap    sw              0       0
>>>> 
>>>> Using instead:
>>>> 
>>>> dumpdev="/dev/ada0s4"
>>>> 
>>>> /dev/ada0s3     /               ufs     rw,noatime      1       1
>>>> /dev/ada0s4     none            swap    sw              0       0
>>>> 
>>>> This allowed the boot to work normally.
>>> 
>>> The use of filemon was tied to buildworld/buildkernel
>>> related activity.
>>> 
>>> But it was kldload that was listed in the crash from
>>> the attempted kldload -n filemon.
>>> 
>>> This would seem similar to having:
>>> 
>>> geom_label_load="YES"
>>> 
>>> in /boot/loader.conf
>> 
>> I added:
>> 
>> device 	filemon
>> device 	geom_label
>> 
>> to the kernel configuration files that
>> I use.
>> 
>> After booting, both work fine, including
>> allowing explicit kldload -n NAME use
>> (that avoids dynamically loading what is
>> already present).
>> 
>> But the boot time use of geom_label is
>> not so lucky in that after booting
>> there is no /dev/label/ or /dev/ufs/ 
>> or such. (No geom_label_load line.)
> 
> I had stupidly forgotten to change
> the /dev/. . . in the likes of
> /etc/fstab , /boot/loader.conf ,
> and /etc/rc.conf to use /dev/label/. . .
> or /dev/ufs/. . .
> 
> When used with the built-in geom_label
> (and no geom_label_load in
> /boot/loader.conf ), it works fine.
> 
>> . . . (junk removed) . . .
>> 
>> But it is also the case that with that
>> line listed in /boot/loader.conf when
>> the kernel has geom_label built-in,
>> powerpc64 FreeBSD still crashes, sort of
>> like geom_label_load does not involve
>> the equivalent of -n in
>> "kld -n geom_label".
>> 
>> . . . (junk removed) . . .
> 
> I updated from -r326192 to -r327075 for
> a TARGET_ARCH=powerpc context and it
> worked fine.
> 
> But this is a gcc 4.2.1 based kernel build
> because with system clang 5 and the system
> binutils the kernel fails to build. (Unlike
> for devel/powerpc64-binutils , there is no
> devel/powerpc-binutils to use in cross
> builds.)

I built world and kernel based on
devel/powerpc64-xtoolchain-gcc and installed
the kernel it produced, leaving world as the
previously installed clang-5.0.0-based build.

kldload does not lead to crashes. Nor
does /boot/loader.conf having:

geom_label_load="YES"

in it.

So my guess is that clang is producing
some form of relocation need that
kldload support does not handle.

I'l note that back at -r326192 this was
not a problem for clang, so the clang
5.0.1 update may be what changed the
status.

===
Mark Millard
markmi at dsl-only.net