how to setup the host interfacce with sriov and mlx5en(4) driver

Hi all,

I am trying to understand how works sriov with the mlx5en(4)  driver.

In particularly should Isetup the VF-0 as passthrough false? On intel NIC i would have to set one of the VF not passthrough for the host. It seems that sr-iov is behaving differently on mellanox: even if all VFs are set with `passthrough=true` I can still configure the interface mce0 on the host.

Is this expected? Does it mean the VF-0 is usable too?

I have setup sr-iov thiis way:
```
$ more /etc/iov/mce0.conf
PF {
device: "mlx5_core0";
num_vfs: 8;
}

DEFAULT {
passthrough: true;
}

# VF used by host
#VF-0 {
# passthrough: false;#}
```

In rc.conf i have the line `iovctl_files="/etc/iov/mce0.conf" ` to enable it.

I can see sriov is workig using `pciconf -lev` which return a numbe rof ppt devices.

````
mlx5_core0@pci0:17:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx]'
class = network
subclass = ethernet
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
mlx5_core1@pci0:17:0:1: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx]'
class = network
subclass = ethernet
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
--
vendor = 'Adaptec'
device = 'Smart Storage PQI SAS'
class = mass storage
subclass = SAS
ppt0@pci0:17:0:2: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt1@pci0:17:0:3: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt2@pci0:17:0:4: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt3@pci0:17:0:5: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt4@pci0:17:0:6: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt5@pci0:17:0:7: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt6@pci0:17:0:8: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network
subclass = ethernet
ppt7@pci0:17:0:9: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1016 subvendor=0x1590 subdevice=0x00d4
vendor = 'Mellanox Technologies'
device = 'MT27710 Family [ConnectX-4 Lx Virtual Function]'
class = network subclass = ethernet
````

I can still set the derive mce0 using ifconfig on the host and I have access to all ppt in bhyve:

host:
```
ifconfig mce0
mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=7eed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO>
ether 9c:dc:71:4c:84:f0
inet6 fe80::9edc:71ff:fe4c:84f0%mce0 prefixlen 64 scopeid 0x4
inet6 2a0e:e701:110b:102::12 prefixlen 64
media: Ethernet 25GBase-SR <full-duplex,rxpause,txpause>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
```

In bhybe I have setup as passthrough: 17/0/3=2:0 so the third ppt.

Benoît