RE: [EXTERNAL] pcib msix allocation in arm64
- In reply to: Souradeep Chakrabarti : "RE: [EXTERNAL] pcib msix allocation in arm64"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Fri, 02 Dec 2022 10:44:14 UTC
> -----Original Message-----
> From: Souradeep Chakrabarti
> Sent: Friday, December 2, 2022 2:50 AM
> To: Andrew Turner <andrew@fubar.geek.nz>; Andrew Turner
> <andrew@freebsd.org>
> Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>; freebsd-
> hackers@FreeBSD.org
> Subject: RE: [EXTERNAL] pcib msix allocation in arm64
>
>
> Adding Andrew.
>
> > -----Original Message-----
> > From: Souradeep Chakrabarti
> > Sent: Thursday, December 1, 2022 5:24 PM
> > To: Andrew Turner <andrew@fubar.geek.nz>
> > Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>;
> freebsd-
> > hackers@FreeBSD.org
> > Subject: RE: [EXTERNAL] pcib msix allocation in arm64
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: Andrew Turner <andrew@fubar.geek.nz>
> > > Sent: Thursday, November 3, 2022 6:21 PM
> > > To: Souradeep Chakrabarti <schakrabarti@microsoft.com>
> > > Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>;
> > freebsd-
> > > hackers@FreeBSD.org
> > > Subject: Re: [EXTERNAL] pcib msix allocation in arm64
> > >
> > > Hi Souradeep,
> > >
> > > For the vmbus_pcib driver you’ll need to implement the
> > > pcib_alloc_msi, pcib_release_msi, and the msix versions, and pcib_map_msi.
> > >
> > > For alloc and release you can just call into the same intr_*
> > > function as pci_host_generic_acpi.c. You’ll need to pass in the xref
> > > for the interrupt
> > controller.
> > > It needs to be the xref the MSI controller registered. For the
> > > GICv2m it’s ACPI_MSI_XREF.
> > [Souradeep]
> > I am currently specifying gic_mbi_start and gic_mbi_end with SPI start
> > and end number, in gic_v3_acpi_attach() and I can see my
> > intr_alloc_msi() and
> > intr_map_msi() are working. But then mlx driver is getting stuck at CREATE_EQ .
> >
> > mlx5_core0: WARN: wait_func:967:(pid 0): CREATE_EQ(0x301) timeout.
> > Will cause a leak of a command resource
> > mlx5_core0: WARN: mlx5_start_eqs:588:(pid 0): failed to create async
> > EQ -60
> > mlx5_core0: ERR: mlx5_load_one:1157:(pid 0): Failed to start pages and
> > async EQs
> > mlx5_core0: WARN: wait_func:967:(pid 0): MANAGE_PAGES(0x108) timeout.
> > Will cause a leak of a command resource
> > mlx5_core0: ERR: reclaim_pages:444:(pid 0): failed reclaiming pages
> > mlx5_core0: WARN: mlx5_reclaim_startup_pages:574:(pid 0): failed
> > reclaiming pages (-60)
> > mlx5_core0: ERR: init_one:1644:(pid 0): mlx5_load_one failed -60
> >
> >
> > gic_v3_acpi_attach( )
> > --
> > --
> > 323 sc->gic_mbi_start = 64;
> > 324 sc->gic_mbi_end = 920;
> > 325
> > 326 err = gic_v3_attach(dev);
> > 327 if (err != 0)
> > 328 goto error;
> > 329
> > 330 sc->gic_pic = intr_pic_register(dev, ACPI_INTR_XREF);
> > 331 if (sc->gic_pic == NULL) {
> > 332 device_printf(dev, "could not register PIC\n");
> > 333 err = ENXIO;
> > 334 goto error;
> > 335 }
> > 336 err = intr_msi_register(dev, ACPI_MSI_XREF);
> > 337 if (err) {
> > 338 device_printf(dev, "could not register MSI\n");
> > 339 }
> > 340 if (intr_pic_claim_root(dev, ACPI_INTR_XREF, arm_gic_v3_intr, sc,
> > 341 GIC_LAST_SGI - GIC_FIRST_SGI + 1) != 0) {
> > 342 err = ENXIO;
> > 343 goto error;
> > 344 }
> > We don't have SPI range mentioned in ACPI DSDT, so I need to specify
> > it manually in the code.
> > It looks like MSI interrupt is not coming to the mlx.
> > Is there any interrupt remapping blocked by FreeBSD kernel here?
> >
> > >
> > > For pcib_map_msi I am unsure if the current code is valid on arm64.
> > > If not you’ll need to call intr_map_msi.
> > >
> > > Andrew
> > >
> > > > On 2 Nov 2022, at 22:27, Souradeep Chakrabarti
> > > > <schakrabarti@microsoft.com>
> > > wrote:
> > > >
> > > > Hi Andrew,
> > > > Thanks for the reply. Regarding generic_pcie_acpi_alloc_msix( ),
> > > > it can be called if the PCI device is child of the generic pcib (
> > > DRIVER_MODULE(pcib, acpi, generic_pcie_acpi_driver, 0, 0) .
> > > > But if the PCI device is communicating with a different pcib
> > > > driver (like vmbus_pcib), in that case do we need to implement all
> > > > these functions of
> > > pci_host_generic_acpi.c ?
> > > >
> > > > Or there are some ways to reuse them?
> > > >
> > > >> -----Original Message-----
> > > >> From: Andrew Turner <andrew@fubar.geek.nz>
> > > >> Sent: Wednesday, November 2, 2022 6:54 PM
> > > >> To: Souradeep Chakrabarti <schakrabarti@microsoft.com>
> > > >> Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>;
> > > >> freebsd- hackers@FreeBSD.org
> > > >> Subject: [EXTERNAL] Re: pcib msix allocation in arm64
> > > >>
> > > >>
> > > >>> On 2 Nov 2022, at 12:56, Souradeep Chakrabarti
> > > >>> <schakrabarti@microsoft.com>
> > > >> wrote:
> > > >>>
> > > >>> Hi,
> > > >>> I can see in x86 nexus.c has implemented pcib_alloc_msix using
> > > >> nexus_alloc_msix().
> > > >>> Which calls msix_alloc() for msix allocation.
> > > >>>
> > > >>> But in case of arm64 I don't find similar pcib_alloc_msix
> > > >>> implementation in
> > > >> nexus.c .
> > > >>> So, on arm64 what is correct way to get allocate msix ?
> > > >>
> > > >> For an arm64 system with ACPI it is most likely handled in
> > > >> generic_pcie_acpi_release_msix. For FDT it can depend on which
> > > >> PCI driver is used.
> > > >>
> > > >> In either case it will call into intr_release_msix that then
> > > >> calls into the MSI controller to allocate the vectors. For a
> > > >> GICv3 driver it will either be gicv3_its_alloc_msix if you have
> > > >> an ITS device, or gic_v3_alloc_msix if using MBI ranges.
> > > >>
> > > >> On ACPI we don’t currently support MBI ranges, although it looks
> > > >> like this could be handled by the existing gicv2m driver. This
> > > >> driver should already work as a child of the GICv3, however it
> > > >> appears to be FDT only, so will need some work to add ACPI support.
> > > >>
> > > >> Andrew
> > > >
[Souradeep]
One more question here, when enabling IOMMU option in arm64, I am getting multiple
compilation errors. Has FreeBSD been compiled with IOMMU for arm64?
In file included from iommu_if.c:31:
./iommu_if.h:173:33: error: unknown type name 'pcell_t'
pcell_t *cells, int ncells);
^
./iommu_if.h:176:39: error: unknown type name 'pcell_t'
pcell_t *cells, int ncells)
^
2 errors generated.
/datadrive/sandbox3/src/sys/arm64/iommu/smmu_fdt.c:208:11: error: too many arguments provided to function-like macro invocation 0, 0, BUS_PASS_INTERRUPT + BUS_PASS_ORDER_MIDDLE); ^ /datadrive/sandbox3/src/sys/sys/bus.h:826:9: note: macro 'EARLY_DRIVER_MODULE' defined here #define EARLY_DRIVER_MODULE(name, busname, driver, evh, arg, pass) \ ^ /datadrive/sandbox3/src/sys/arm64/iommu/smmu_fdt.c:207:1: error: type specifier missing, defaults to 'int' [-Werror,-Wimplicit-int] EARLY_DRIVER_MODULE(smmu, simplebus, smmu_fdt_driver, smmu_fdt_devclass, ^