Re: enabling same PPI interrupt to all CPU in ARM64 SMP

On Fri, May 12, 2023 at 9:51 AM Souradeep Chakrabarti
<schakrabarti@microsoft.com> wrote:
>
>
>
>
> >-----Original Message-----
> >From: Souradeep Chakrabarti
> >Sent: Monday, May 8, 2023 6:39 PM
> >To: Kyle Evans <kevans@freebsd.org>
> >Cc: Wei Hu <weh@microsoft.com>; freebsd-hackers@FreeBSD.org
> >Subject: enabling same PPI interrupt to all CPU in ARM64 SMP
> >
> >Hi ,
> >
> >While using SMP in ARM64 Hyper-V we are getting stuck in boot if there is a
> >interrupt for VMBus coming to CPU1 and VMBus interrupt handler is not getting
> >that interrupt.
> >
> >In ARM64 Hyper-V we are using IRQ18 for VMBus and it is a PPI interrupt.
> >
> >But Hypev-V host sends interrupt to this IRQ 18 for both CPU0 and CPU1 in 2CPU
> >system.
> >This is based on the corresponding VMBus channel which assigned with the CPU.
> >
> >Now VMBus ISR is getting the interrupt in CPU0 but not getting from CPU1.
> >Any idea, how we can use the same PPI 18 for all the CPU cores?
> >
> >Any help will be appreciated, as this is blocking the enablement of FreeBSD in Azure
> >ARM64.
> [Souradeep]
> Can someone please help me it.
>

Looking at least at the GIC implementation, it looks like this is a
known limitation:

 875         /*
 876          * XXX - In case that per CPU interrupt is going to be
enabled in time
 877          *       when SMP is already started, we need some IPI
call which
 878          *       enables it on others CPUs. Further, it's more
complicated as
 879          *       pic_enable_source() and pic_disable_source()
should act on
 880          *       per CPU basis only. Thus, it should be solved
here somehow.
 881          */
 882         if (isrc->isrc_flags & INTR_ISRCF_PPI)
 883                 CPU_SET(PCPU_GET(cpuid), &isrc->isrc_cpu);

I think we need something /like/ this:
https://people.freebsd.org/~kevans/ppi.diff, though it still has the
caveat that PPIs effectively cannot be fully setup before SI_SUB_SMP.
So, it's likely almost a NOP for existing platforms (will emit a
warning with bootverbose for armv8 timers) but might do the trick for
you.

Thanks,

Kyle Evans