RE: [EXTERNAL] pcib msix allocation in arm64

> -----Original Message-----
> From: Andrew Turner <andrew@fubar.geek.nz>
> Sent: Monday, December 5, 2022 8:44 PM
> To: Souradeep Chakrabarti <schakrabarti@microsoft.com>
> Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>; freebsd-
> hackers@FreeBSD.org
> Subject: Re: [EXTERNAL] pcib msix allocation in arm64
> 
> 
> 
> > On 1 Dec 2022, at 11:54, Souradeep Chakrabarti <schakrabarti@microsoft.com>
> wrote:
> >
> >
> >
> >
> >> -----Original Message-----
> >> From: Andrew Turner <andrew@fubar.geek.nz>
> >> Sent: Thursday, November 3, 2022 6:21 PM
> >> To: Souradeep Chakrabarti <schakrabarti@microsoft.com>
> >> Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>;
> >> freebsd- hackers@FreeBSD.org
> >> Subject: Re: [EXTERNAL] pcib msix allocation in arm64
> >>
> >> Hi Souradeep,
> >>
> >> For the vmbus_pcib driver you’ll need to implement the
> >> pcib_alloc_msi, pcib_release_msi, and the msix versions, and pcib_map_msi.
> >>
> >> For alloc and release you can just call into the same intr_* function
> >> as pci_host_generic_acpi.c. You’ll need to pass in the xref for the interrupt
> controller.
> >> It needs to be the xref the MSI controller registered. For the GICv2m
> >> it’s ACPI_MSI_XREF.
> > [Souradeep]
> > I am currently specifying gic_mbi_start and gic_mbi_end with SPI start
> > and end number, in gic_v3_acpi_attach() and I can see my
> > intr_alloc_msi() and intr_map_msi() are working. But then mlx driver is getting
> stuck at CREATE_EQ .
> >
> > mlx5_core0: WARN: wait_func:967:(pid 0): CREATE_EQ(0x301) timeout.
> > Will cause a leak of a command resource
> > mlx5_core0: WARN: mlx5_start_eqs:588:(pid 0): failed to create async
> > EQ -60
> > mlx5_core0: ERR: mlx5_load_one:1157:(pid 0): Failed to start pages and
> > async EQs
> > mlx5_core0: WARN: wait_func:967:(pid 0): MANAGE_PAGES(0x108) timeout.
> > Will cause a leak of a command resource
> > mlx5_core0: ERR: reclaim_pages:444:(pid 0): failed reclaiming pages
> > mlx5_core0: WARN: mlx5_reclaim_startup_pages:574:(pid 0): failed
> > reclaiming pages (-60)
> > mlx5_core0: ERR: init_one:1644:(pid 0): mlx5_load_one failed -60
> >
> >
> > gic_v3_acpi_attach( )
> > --
> > --
> > 323     sc->gic_mbi_start = 64;
> > 324     sc->gic_mbi_end = 920;
> > 325
> > 326     err = gic_v3_attach(dev);
> > 327     if (err != 0)
> > 328         goto error;
> > 329
> > 330     sc->gic_pic = intr_pic_register(dev, ACPI_INTR_XREF);
> > 331     if (sc->gic_pic == NULL) {
> > 332         device_printf(dev, "could not register PIC\n");
> > 333         err = ENXIO;
> > 334         goto error;
> > 335     }
> > 336     err = intr_msi_register(dev, ACPI_MSI_XREF);
> > 337     if (err) {
> > 338         device_printf(dev, "could not register MSI\n");
> > 339     }
> > 340     if (intr_pic_claim_root(dev, ACPI_INTR_XREF, arm_gic_v3_intr, sc,
> > 341         GIC_LAST_SGI - GIC_FIRST_SGI + 1) != 0) {
> > 342         err = ENXIO;
> > 343         goto error;
> > 344     }
> > We don't have SPI range mentioned in ACPI DSDT, so I need to specify it
> manually in the code.
> > It looks like MSI interrupt is not coming to the mlx.
> > Is there any interrupt remapping blocked by FreeBSD kernel here?
> 
> Do you see the expected interrupt in arm_gic_v3_intr? If so it would be useful to
> trace it though intr_isrc_dispatch. If no it may be because the interrupt is not
> being triggered, or is masked.
[Souradeep] 
Thanks for the help. Looks like the problem is from the hardware side, not from OS side.
It seems to be working in Azure. 
> 
> Andrew