bhyve passthrough of SR-IOV

[Chuck Tuffli]

I'm new to the world of SR-IOV and could use some guidance. Does anyone
have experience doing PCI passthrough of SR-IOV virtual functions on
Mellanox CNA's to bhyve? Are there any known limitations? For example, does
RDMA not work?

The reason I ask is, I have a setup running FreeBSD 12-stable with a
Mellanox ConnectX-5 configured to support 16 VFs on each port. All VFs are
configured to be passthrough. If I pass through a physical function to the
Linux VM, the RDMA-based application works correctly. But if I pass through
a virtual function, the RDMA-based application starts complaining about
"ibv_context" (i.e. RDMA) failures. It's worth noting that even in this
case, I can assign an IP address to the device in the VM and ping addresses
on the other side of the wire. So I believe passthrough (mostly?) worked.
Any suggestions? TIA!

--chuck