FreeBSD on Layerscape/QorIQ LX2160X

[Dan Kotowski]

> > > Well, normally PCIe devices can address all the space, unless you have terribly quirky hardware (on the RPi4, PCIe can address 3GB only..)
> > > Also I would guess that like.. yes, even if PCIe goes through the SMMU, if the SMMU is untouched by the OS, interrupts should arrive where the OS expects it normally.
> > >
> > > > And section 4.1.6 System MMU and Device Assignment
> > > > """
> > > > If a device is assigned and passed through to an operating system under a hypervisor, then the memory
> > >
> > > We're running on bare metal, this is about VMs.
> >
> > So even though that stub only returns ENXIO, it should not matter unless I intend to use bhyve or something like that?
> >
> > > > I'm almost surprised that nobody has bumped into this before. How does the IORT look on the MACCHIATObin if interrupts are NOT mapped behind SMMU?
> > >
> > > There is no IORT on the mcbin.. armada8k has a GICv2 not v3 :D
> >
> > I'm also interested in acquiring one of those as well at some point, but your posts indicate that the net ifaces on that are basically dead weight under FreeBSD as well :(
>
> For a desktop, USB Ethernet is enough :D
>
> > > Socionext SynQuacer has the PCI node pointed to the ITS node.
> >
> > 24x A53 cores... hmmm...
>
> Yeah, very low single core perf, especially for that price :/
>
> > > If I'm reading the table disassembly correctly, the Ampere eMAG has PCI nodes pointing to SMMU (v1/v2). But FreeBSD works there fine!
> > > That means the "fallback" non-IORT calculation of RIDs works fine there. But maybe it doesn't on the NXP chip because it's buggy.
> > > But there was some way to get the interrupts (without supporting SMMU) on this NXP chip – as evidenced by NetBSD working!
> > > Again.. can you flash the firmware where PCIe was working under NetBSD, dump the IORT there and also try patched (with D25179, e.g. any of my recent builds I guess) FreeBSD there?
> >
> > https://gist.github.com/agrajag9/a59b6c440365745c5a4036e0faf6e23c
>
> Yeah, so the "old" does have "Output reference : 0x30" under the PCIe roots.

For my own edification, let me know if I'm reading this all correctly.

>From DEN0049D:

"A reference to the output IORT Node.  This field contains the address offset of the IORT Node relative to the start of the IORT. For example, if this ID mapping is for a root complex outputting to an SMMU, the value of this
field isthe difference between the start ofthe SMMU IORTnode and the start
of the IORT."

In the old firmware, under the Root Complex Nodes, `Output reference: 0x30` points to the ITS Node with node offset 0x30.

Then in the new firmware, under the Root Copmlex Nodes, `Output reference:
0x48` points to the SMMU with node offset 0x48.

Is this what you meant when you said everything is "behind the SMMU"?

> So does NetBSD PCIe only work on "old"?

Correct.

> FreeBSD still not working anywhere?

Also correct :(

> I guess another interesting thing to check would be Linux configured without any SMMU support..

Probably a good test! I'll see if I can make a build without it.

As for building my own world...

What patches are we applying right now? I think some new builds would be good, including one with all the stuff we've fixed - like AHCI - but with the NXP PCIe code so I can test on the old firmware. If I'm keeping track correctly, this includes:

- D20835: enable tagged pointers
- D20974: Port sbsawdt drive from NetBSD
- D21017: armv8crypto: add AES-XTS support
- D24423: arm/pmu: add ACPI attachment, more FDT names
- D25145: acpi_resource: support multiple IRQs
- D25157: ahci_generic: add quirk for NXP0004
- D25179: acpi_iort: fix mapping end calculation

Any others?