Re: How to make FreeBSD's kernel boot a RPi4B with modern RPi* firmware

On Jan 7, 2023, at 00:37, Klaus Küchemann <maciphone2@googlemail.com> wrote:

Hi Mark,

> `ve tested your "early_dma“ patch now on the CM4(not tested on the 4b)…
> true, it can boot the latest firmware but after investigating in what’s new 
> in bcm2711-rpi-cm4.dtb I guess new features are mainly  related to CPU (L2-) caches features ,
> That’s why we get things like : "clk_fixed4: <Fixed clock> disabled on ofwbus0
> clk_fixed4: Cannot FDT parameters.“ ...
> so I think we currently won’t benefit much of the new firmware.

But being able to boot with the firmware should make it
easier to deal with investigating other things related
to more modern firmware. Also, it avoids running into
the specific type of crash and having to figure out what
is going on for it: one less problem to wade through.

In fact, the modern firmware corrects mistakes in the
.dtb's relative to the RPi4B PCIe description. For a long
time different parts of the information were inconsistent
with each other. But various OS's (including FreeBSD)
ignore parts of the Device Tree information and do their
own thing independent of parts of the Device Tree
information present. This allowed various things to work
even when ignoring things was not a deliberate way to
avoid the inconsistency(s).

FreeBSD may well have hardcoded RPi4B specifics that would
be wrong for a CM4. But it is still better to target
getting the CM4 going via using the corrected Device Tree
information via targeting a more recent RPi* firmware
vintage that has the fixed information --instead of
targeting the old vintage with the known mistakes in the
Device Tree.

> I had a little hope that an earlier dma could shine a bit more light on bugs like this : 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260131

The only BCM2711's that I've access to are RPi4B's so I do
not see such problems. I've no clue about them.

> But a tiny little hope can’t of course fix bugs :-) , so I think we have to focus more on fixing existing bugs and 
> Adding device drivers(of course the ToDo-list is not news but still the truth;-)
> 
> Thanks for taking attention the current firmware things, always worth to take a look into.

Thanks for testing that the bcm_dma related patching
applies to avoiding the specific crash type in more
than my particular contexts.

This weekend I'm trying to upgrade all the RPi4B boot
media that I use to be using modern RPi* firmware
(without boot crashes) as part of a general round of
updating the FreeBSD version things are based on.
(Also: RPi3B, RPi2B Rev 1.2, and [armv7] RPi2B Rev 1.1
boot media.)

> Regards
> 
> K.
> 
>> Am 25.12.2022 um 17:37 schrieb Mark Millard <marklmi@yahoo.com>:
>> 
>> On Dec 24, 2022, at 20:14, Mark Millard <marklmi@yahoo.com> wrote:
>> 
>>> On Dec 24, 2022, at 19:15, Mark Millard <marklmi@yahoo.com> wrote:
>>> 
>>>> I finally ran into EARLY_DRIVER_MODULE, BUS_PASS_RESOURCE,
>>>> BUS_PASS_ORDER_MIDDLE and the like and they allow being
>>>> sure that the brcm,bcm2835-dma related setup has been done
>>>> before any use of it is made, despite the order in the
>>>> Device Tree: use an earlier pass for brcm,bcm2835-dma
>>>> related attach. This avoids the kernel crashing during
>>>> boot.
>>>> 
>>>> The example context used below has: serial console with
>>>> USB3 SSD boot media (not requiring a usb_pgood_delay
>>>> setting), booting a stable/13. The RPI4B is a C0T one (no
>>>> 3 GiByte limitation, for example: the PCIe wrapper logic
>>>> has been corrected).
>>>> 
>>>> stable/13's source code changes ( similarly for
>>>> releng/13.1 ):
>>>> 
>>>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>> index cab8639bb607..d8b49acfe332 100644
>>>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>> @@ -766,5 +766,6 @@ static driver_t bcm_dma_driver = {
>>>> 
>>>> static devclass_t bcm_dma_devclass;
>>>> 
>>>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, bcm_dma_devclass, 0, 0);
>>>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, bcm_dma_devclass,
>>>> +    0, 0, BUS_PASS_RESOURCE + BUS_PASS_ORDER_MIDDLE);
>>>> MODULE_VERSION(bcm_dma, 1);
>>>> 
>>>> 
>>>> For reference, a 13S snapshot with my kernel build replacing
>>>> the snapshot's kernel, was booted with:
>>>> 
>>>> # strings /boot/msdos/start4.elf | grep VC_BUILD_ID_
>>>> VC_BUILD_ID_USER: dom
>>>> VC_BUILD_ID_TIME: 11:09:05
>>>> VC_BUILD_ID_VARIANT: start
>>>> VC_BUILD_ID_TIME: Oct 26 2022
>>>> VC_BUILD_ID_BRANCH: bcm2711_2
>>>> VC_BUILD_ID_HOSTNAME: buildbot
>>>> VC_BUILD_ID_PLATFORM: raspberrypi_linux
>>>> VC_BUILD_ID_VERSION: c72ad6b26ff40c91ef776b847436094ee63fabee (clean)
>>>> 
>>>> There are new things present that FreeBSD reports
>>>> but ignores, producing messages like:
>>>> 
>>>> clk_fixed4: <Fixed clock> disabled on ofwbus0
>>>> clk_fixed4: Cannot FDT parameters.
>>>> device_attach: clk_fixed4 attach returned 6
>>>> 
>>>> over and over during part of the boot. It seems to
>>>> retry as it goes and thus produce so many messages.
>>>> 
>>>> There was also:
>>>> 
>>>> fb0: <BCM2835 VT framebuffer driver> on simplebus0
>>>> fb0: changing fb bpp from 0 to 24
>>>> mbox0: mbox response error
>>>> fb0: bcm2835_mbox_fb_init failed, err=5
>>>> device_attach: fb0 attach returned 6
>>>> 
>>>> genet0 is working.
>>>> 
>>>> I've not checked if the microsd card slot can be used.
>>>> 
>>>> I used the normal FreeBSD U-Boot since I was not booting
>>>> the NVM3 media that requires extra time (usb_pgood_delay
>>>> would be assigned in my own U-Boot builds).
>>>> 
>>>> For reference, I used my typical sort of config.txt :
>>>> 
>>>> # more /boot/msdos/config.txt
>>>> [all]
>>>> arm_64bit=1
>>>> dtparam=audio=on,i2c_arm=on,spi=on
>>>> dtoverlay=mmc
>>>> dtoverlay=disable-bt
>>>> device_tree_address=0x4000
>>>> kernel=u-boot.bin
>>>> 
>>>> [pi4]
>>>> hdmi_safe=1
>>>> armstub=armstub8-gic.bin
>>>> 
>>>> #
>>>> [all]
>>>> #
>>>> # Local addition that avoids USB3 SSD boot failures that look like:
>>>> #   uhub_reattach_port: port ? reset failed, error=USB_ERR_TIMEOUT
>>>> #   uhub_reattach_port: device problem (USB_ERR_TIMEOUT), disabling port ?
>>>> initial_turbo=60
>>>> # U-Boot that has, for example, a built-in usb_pgood_delay assignment
>>>> # for a media specific issue added:
>>>> #kernel=u-boot.bin.2022.10.arm64
>>>> #
>>>> # Local additions:
>>>> enable_uart=1
>>>> uart_2ndstage=1
>>>> dtdebug=1
>>>> disable_commandline_tags=1
>>>> disable_overscan=1
>>>> #gpu_mem_1024=32
>>>> #
>>>> #program_usb_boot_mode=1
>>>> #program_usb_boot_timeout=1
>>>> 
>>>> # Old RPi3's/RPi2Bv1.2's may ignore [pi4] and the like.
>>>> # That would make the below inappropriate for such contexts.
>>>> [pi4]
>>>> # Locally avoid hdmi_safe's dislay scaling:
>>>> hdmi_safe=0
>>>> #
>>>> armstub=armstub8-gic.bin
>>>> #
>>>> # Local additions:
>>>> over_voltage=6
>>>> arm_freq=2000
>>>> sdram_freq_min=3200
>>>> force_turbo=1
>>>> #
>>>> #total_mem=1024
>>>> #total_mem=991
>>>> [all]
>>>> 
>>>> [pi3] 
>>>> armstub=armstub8.bin
>>>> dtoverlay=pwm
>>>> audio_pwm_mode=0
>>>> [all]
>>>> 
>>>> 
>>>> As for main [so: 14], the devclass_t use is gone, thus
>>>> the source code changes are:
>>>> 
>>>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>> index 5f9ecb0b7981..83c4c493a66b 100644
>>>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>>>> @@ -764,5 +764,6 @@ static driver_t bcm_dma_driver = {
>>>>     sizeof(struct bcm_dma_softc),
>>>> };
>>>> 
>>>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0);
>>>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0,
>>>> +    BUS_PASS_RESOURCE + BUS_PASS_ORDER_MIDDLE);
>>>> MODULE_VERSION(bcm_dma, 1);
>>>> 
>>>> 
>>> 
>>> I should note that the above is not likely to be
>>> the most appropriate in detail. The boot reports:
>>> 
>>> # dmesg -a | grep bcm_dma0
>>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>>> bcm_dma0: cannot allocate interrupt
>>> device_attach: bcm_dma0 attach returned 6
>>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>>> bcm_dma0: cannot allocate interrupt
>>> device_attach: bcm_dma0 attach returned 6
>>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>>> bcm_dma0: cannot allocate interrupt
>>> device_attach: bcm_dma0 attach returned 6
>>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>>> bcm_dma0: cannot allocate interrupt
>>> device_attach: bcm_dma0 attach returned 6
>>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>>> 
>>> where that last (working) one has the message
>>> context:
>>> 
>>> gic0: <ARM Generic Interrupt Controller> mem 0x40041000-0x40041fff,0x40042000-0x40043fff,0x40044000-0x40045fff,0x40046000-0x40047fff irq 30 on simplebus0
>>> gic0: pn 0x2, arch 0x2, rev 0x1, implementer 0x43b irqs 256
>>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq 31,32,33,34,35,36,37,38,39,40,41 on simplebus0
>>> 
>>> So something involving BUS_PASS_INTERRUPT or later
>>> (but before, say, BUS_PASS_SUPPORTDEV) may be more
>>> appropriate. Possibly:
>>> 
>>> BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE
>>> 
>>> (so after gic0).
>>> 
>>> 
>> 
>> So, I'm now using . . .
>> (leading whitespace possibly not accurately preserved)
>> 
>> 
>> stable/13's source code changes are ( similarly for
>> releng/13.1 ):
>> 
>> # git -C /usr/13S-src/ diff sys/arm/broadcom/bcm2835/bcm2835_dma.c
>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>> index cab8639bb607..6d521d6dcace 100644
>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>> @@ -766,5 +766,6 @@ static driver_t bcm_dma_driver = {
>> 
>> static devclass_t bcm_dma_devclass;
>> 
>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, bcm_dma_devclass, 0, 0);
>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, bcm_dma_devclass,
>> +    0, 0, BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE);
>> MODULE_VERSION(bcm_dma, 1);
>> 
>> 
>> main's [so: 14's] source code changes are:
>> 
>> # git -C /usr/main-src/ diff sys/arm/broadcom/bcm2835/bcm2835_dma.c
>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>> index 5f9ecb0b7981..d901447df1e9 100644
>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c
>> @@ -764,5 +764,6 @@ static driver_t bcm_dma_driver = {
>>        sizeof(struct bcm_dma_softc),
>> };
>> 
>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0);
>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0,
>> +    BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE);
>> MODULE_VERSION(bcm_dma, 1);
>> 


===
Mark Millard
marklmi at yahoo.com