RE: [EXTERNAL] pcib msix allocation in arm64

> -----Original Message-----
> From: Souradeep Chakrabarti
> Sent: Thursday, November 10, 2022 3:20 PM
> To: Andrew Turner <andrew@fubar.geek.nz>
> Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>; freebsd-
> hackers@FreeBSD.org
> Subject: RE: [EXTERNAL] pcib msix allocation in arm64
> 
> 
> 
> 
> > -----Original Message-----
> > From: Andrew Turner <andrew@fubar.geek.nz>
> > Sent: Thursday, November 3, 2022 6:21 PM
> > To: Souradeep Chakrabarti <schakrabarti@microsoft.com>
> > Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>;
> freebsd-
> > hackers@FreeBSD.org
> > Subject: Re: [EXTERNAL] pcib msix allocation in arm64
> >
> > Hi Souradeep,
> >
> > For the vmbus_pcib driver you’ll need to implement the pcib_alloc_msi,
> > pcib_release_msi, and the msix versions, and pcib_map_msi.
> >
> > For alloc and release you can just call into the same intr_* function
> > as pci_host_generic_acpi.c. You’ll need to pass in the xref for the
> > interrupt controller. It needs to be the xref the MSI controller
> > registered. For the GICv2m it’s ACPI_MSI_XREF.
> >
> [Souradeep]
> I have used following in vmbus_pcib.c :
> ret = intr_alloc_msix(pcib, dev, ACPI_MSI_XREF, irq); But it is failing in
> pic_lookup() with ESRCH by not finding pic with ACPI_MSI_XREF.
> Do I need to do anything for pic_lookup() to be successful?
> 
[Souradeep] 
I found out the reason here pic_lookup() for ACPI_MSI_XREF is failing, as
we are not calling intr_msi_register() but intr_pic_register() is getting called for the PCIB.
This is because we don't have ITS implemented in Hyper-V. Instead, Hyper-V is presenting the
MSI/MSI-X as SPI because of it's own limitation.
Linux has solved it using custom irqchip driver to handle it. 
But I am not sure how to address it in FreeBSD. 
Can you please point me out something here?
> > For pcib_map_msi I am unsure if the current code is valid on arm64. If
> > not you’ll need to call intr_map_msi.
> >
> > Andrew
> >
> > > On 2 Nov 2022, at 22:27, Souradeep Chakrabarti
> > > <schakrabarti@microsoft.com>
> > wrote:
> > >
> > > Hi Andrew,
> > > Thanks for the reply. Regarding generic_pcie_acpi_alloc_msix( ), it
> > > can be called if the PCI device is child of the generic pcib (
> > DRIVER_MODULE(pcib, acpi, generic_pcie_acpi_driver, 0, 0) .
> > > But if the PCI device is communicating with a different pcib driver
> > > (like vmbus_pcib), in that case do we need to implement all these
> > > functions of
> > pci_host_generic_acpi.c ?
> > >
> > > Or there are some ways to reuse them?
> > >
> > >> -----Original Message-----
> > >> From: Andrew Turner <andrew@fubar.geek.nz>
> > >> Sent: Wednesday, November 2, 2022 6:54 PM
> > >> To: Souradeep Chakrabarti <schakrabarti@microsoft.com>
> > >> Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>;
> > >> freebsd- hackers@FreeBSD.org
> > >> Subject: [EXTERNAL] Re: pcib msix allocation in arm64
> > >>
> > >>
> > >>> On 2 Nov 2022, at 12:56, Souradeep Chakrabarti
> > >>> <schakrabarti@microsoft.com>
> > >> wrote:
> > >>>
> > >>> Hi,
> > >>> I can see in x86 nexus.c has implemented pcib_alloc_msix using
> > >> nexus_alloc_msix().
> > >>> Which calls msix_alloc() for msix allocation.
> > >>>
> > >>> But in case of arm64 I don't find similar pcib_alloc_msix
> > >>> implementation in
> > >> nexus.c .
> > >>> So, on arm64 what is correct way to get allocate msix ?
> > >>
> > >> For an arm64 system with ACPI it is most likely handled in
> > >> generic_pcie_acpi_release_msix. For FDT it can depend on which PCI
> > >> driver is used.
> > >>
> > >> In either case it will call into intr_release_msix that then calls
> > >> into the MSI controller to allocate the vectors. For a GICv3 driver
> > >> it will either be gicv3_its_alloc_msix if you have an ITS device,
> > >> or gic_v3_alloc_msix if using MBI ranges.
> > >>
> > >> On ACPI we don’t currently support MBI ranges, although it looks
> > >> like this could be handled by the existing gicv2m driver. This
> > >> driver should already work as a child of the GICv3, however it
> > >> appears to be FDT only, so will need some work to add ACPI support.
> > >>
> > >> Andrew
> > >