vRSS support on FreeBSD

[Adrian Chadd]

On 11 August 2014 02:48, Wei Hu <weh at microsoft.com> wrote:
> CC freebsd-net@ for wider discussion.
>
> Hi Adrian,
>
> Many thanks for the explanation.  I checked the if_igb.c  and found the flowid field was set in the RX side in igb_rxeof():
>
> Igb_rxeof()
> {
>  ...
> #ifdef  RSS
>                         /* XXX set flowtype once this works right */
>                         rxr->fmp->m_pkthdr.flowid =
>                             le32toh(cur->wb.lower.hi_dword.rss);
>                         rxr->fmp->m_flags |= M_FLOWID;
>  ...
> }
>
> I have two questions regarding this.
>
> 1. Is the RSS hash value stored in cur->wb.lower.hi_dword.rss set by the NIC hardware?

Yup.

> 2. So the hash value and m_flags are stored in the mbuf related to the received packet on the rx side(lgb_rxeof()). But we check the hash value and m_flags in mbuf related to the send packet on the tx side (in igb_mq_start()). Does the kernel re-use the same mbuf for tx? If so, how does it know for the same network stream it should use the same mbuf got from the rx for packet sending? If not, how does the kernel preserve the same hash value across the rx mbuf and tx mbuf for same network stream? This seems quite magical to me.

The mbuf flowid/flowtype ends up in the inpcb->inp_flowid /
inpcb->inp_flowtype as part of the TCP receive path.

Then whenever the TCP code outputs an mbuf, it copies the inpcb flow
details out to outbound mbufs.

>
> For the Hyper-V case, the host controls which vCPU it wants to interrupt. And the rule can change dynamically based on the load. For a non-busy VM, host will send most packets to same vCPU for power saving purpose. For a busy VM, host will distribute the packets evenly across all vCPUs. This means host could change the RSS bucket mapping dynamically. Hyper-V does this by sending a mapping table to VM whenever the it needs update. This also means we cannot use FreeBSD's own bucket mapping which I believe is fixed. Also Hyper-V use its own hash key. So do you think it is possible we still use the exisiting RSS infrastructure built in FreeBSD in this purpose?

Eventually. Doing rebalancing in RSS is on the TODO list, after I get
the rest of the basic packet handling / routing done.

How's vRSS notify the VM that the mapping table has changed? What's
the format of it look like?


-a