Is ixgbe usable with SR-IOV?

[Colin Percival]

Hi Xen people & jfv,

Amazon's new "C3" instance type has support for SR-IOV using ixgbe hardware;
our driver looks like it should support this, but when I turn it on it doesn't
seem to work.  Boot dmesg: http://pastebin.com/1Kkpfzbi

The virtual hardware is recognized:
> ix0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.1.4> mem 0xf3000000-0xf3003fff,0xf3004000-0xf3007fff at device 3.0 on pci0
> ix0: Using MSIX interrupts with 2 vectors
> ix0: Ethernet address: 02:91:23:76:38:77

And we seem to be able to send packets, but never see any responses:
> ix0: link state changed to UP
> Starting Network: lo0 ix0.
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>         options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
>         inet6 ::1 prefixlen 128
>         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
>         inet 127.0.0.1 netmask 0xff000000
>         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
>         ether 02:91:23:76:38:77
>         nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>         media: Ethernet autoselect
>         status: active
> Starting devd.
> Starting dhclient.
> DHCPDISCOVER on ix0 to 255.255.255.255 port 67 interval 8
> DHCPDISCOVER on ix0 to 255.255.255.255 port 67 interval 12
> DHCPDISCOVER on ix0 to 255.255.255.255 port 67 interval 12
> DHCPDISCOVER on ix0 to 255.255.255.255 port 67 interval 9
> DHCPDISCOVER on ix0 to 255.255.255.255 port 67 interval 10
> DHCPDISCOVER on ix0 to 255.255.255.255 port 67 interval 10
> No DHCPOFFERS received.

Under exactly the same Xen configuration except with SR-IOV turned off and
EC2 presenting a Xen netback to us instead, the DHCP works just fine.

Has anyone managed to use {FreeBSD, SR-IOV, ixgbe, Xen}?  Was any magic
necessary in order to make it work?

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid