msi-x enabled igb works only if module loaded twice [Was: Re: kldload if_igb twice needed to bring nic into operation]

[Harald Schmalzbauer]

 schrieb Jack Vogel am 23.10.2012 23:25 (localtime):
> LOL, wow this is interesting. When I first was developing the VF support,

Well, in fact I choose 'kawela' (82576) because I originally wanted to
use VFs. But I can't get SR-IOV working with ESXi5.1.
I'm using async-drivers, and I have option "max_vfs" available and
enabled, also at boot time ESXi detects SR-IOV:
    cpu0:4096)PCI: 5327: 00:02:00.0: Found Single Root I/O
Virtualization support
But later on, I get this error:
    cpu0:4573)<6>igb: : igb_validate_option: max_vfs - SR-IOV VF devices
set to 4
    cpu0:4573)<4>igb 0000:02:00.0: Failed to initialize SR-IOV
virtualization

Since I have no VMware support contract and found absoluetly no
documentation, I gave up and plugged in a second kawela for passing
through into my iSCSI-target-guest.
I found some hints that SR-IOV needs additional BIOS features besides
VT-x and VT-d, because SR-IOV cards need to allocate additional address
space. But that's beyond my PCI knowledge, and my BearTooth board
doesn't have such SR-IOV switches, so I'm trying to forget that I've
ever heard of SR-IOV :-(

Btw, are there plans to support IPSec offload for kawela?

...
> There is nothing you can do about this because the issue is in the host,
> not the guest,
> well getting the host code fixed is the solution :)
>
> Hope this helps,

Thanks a lot, VMware has to fix more passthrough issues, since 5.1
crashes easily with passthrough-setups and many people report PODs (pink
screen of death; I've first seen that with 5.1)
But unfortunately there's no way for me to get in contact with VMWare to
raise this MSI-X-passthrough topic. This would porbably also fix the
mps-msix-problems, but disabling MSI-X for mps doesn't cost that much
interrupts. With igb I see 25k irq/s difference between MSI and MSI-X. I
don't understand why, but it's too expensive for my single-socket system.

Thanks a lot,

-Harry


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20121024/2d97d2f6/attachment.sig>