PCIe passthrough really that expensive?

[Anish]

Hi Harry,
>Any hints highly appreciated!
In my setup, I have dual port Intel GigE, one assigned to host and other
one is used by guest

root at svmhost:~ # pciconf -l |grep ppt

...

ppt2 at pci0:2:0:0: class=0x020000 card=0x125e8086 chip=0x105e8086 rev=0x06
hdr=0x00

root at svmhost:~ #


This show up as 'em0' in BSD guest with 2 vcpus.

root at bsdguest:~ # pciconf -l |grep em0

em0 at pci0:0:21:0:        class=0x020000 card=0x125e8086 chip=0x105e8086
rev=0x06 hdr=0x00

root at bsdguest:~ #

Once guest is booted, ppt2 will claim interrupt resources, in this case
just 1 interrupt line #265

root at svmhost:~ # vmstat -i

interrupt                          total       rate

...

irq263: em0:irq0                 1028705        634

..

irq265: ppt2                     1041187        641

Total                            2835121       1747

Now use cpuset to route IRQ#265 to say core 0

$cpuset -l 0 -x 265

Again use cpuset to force VM[PID 1222] to run on all core except #0

root at svmhost:~ # ps

 PID TT  STAT    TIME COMMAND

....

1222  1  I+   5:59.45 bhyve: vm1 (bhyve)


VM can run on all cores except #0.

$ cpuset -l 1-3 -p 1222


You can monitor guest due to interrupts using

$root at svmhost:~ # bhyvectl --get-stats --vm=<vm name> --cpu=<vcpu> | grep
external

vm exits due to external interrupt      27273

root at svmhost:~ #


Regards,

Anish

On Sun, Jun 11, 2017 at 2:51 AM, Harry Schmalzbauer <freebsd at omnilan.de>
wrote:

>  Bezüglich Harry Schmalzbauer's Nachricht vom 09.06.2017 10:22 (localtime):
> > Bezüglich Anish's Nachricht vom 08.06.2017 14:35 (localtime):
> >> Hi Harry,
> >>> I thought I'd save these expensive VM_Exits by using the passthru path.
> >> Completely wrong, is it?
> >>
> >> It depends on which processor you are using. For example APICv was
> >> introduced in IvyBridge which enabled h/w assisted localAPIC rather than
> >> using s/w emulated, bhyve supports it on Intel processors.
> …
> > I'm still usign IvyBridge (E3v2) with this "new" machine, but haven't
> > ever heard/thought about APCIv!
>
> It seems APICv is available on IvyBridge-EP (Xeon E5/E7v2) only, not for
> E3v2 :-(
> Furthermore, if I didn't miss anything in the datasheets, no currently
> available E3 Xeon offers local APIC virtualization. Can somebody of the
> xperts confirm that?
>
>
> …
> >> Can you run a simple experiment, assign pptdev interrupts to core that's
> >> not running guest/vcpu? This will reduce #VMEXIT on vcpu which we know
> >> is expensive.
> > Interesting approach.  But I have no idea how I should assign a PCIe
> > specific core to a PCIe dev.  Is it pptdev specific? The tunables in
> > device.hints(5) can't be used for that, can they?
>
> I wasn't able to find out how to do that.
> Any hints highly appreciated!
>
> -harry
>
>