vRSS support on FreeBSD

[Adrian Chadd]

Hi!

Is there a spec for this stuff floating around somewhere?

What do other platforms do for receive/transmit affinity on hyperv?



-a


On 12 August 2014 21:08, Wei Hu <weh at microsoft.com> wrote:
> Hi Adrian,
>
> The send mapping table is an array with fixed the size of elements, say VRSS_TAB_SIZE. It contains the tx queue number on which TX packet should be sent. So the vCPU = Send_table[hash-value % VRSS_TAB_SIZE % number_of_tx_queue] is the way to choose the tx queue. Send_table is updated by the host every few minutes (on a busy system) or hours (on a light system).
>
> Since the vNIC doesn't give guest VM the hash value for a rx packet, I am thinking maybe I can put the rx queue number in the m_pkthdr.flowid of the mbuf on the receiving path. So the queue number will be passed to the mbuf on the sending path. This way we choose the same queue to send the packet, and we don't need to calculate the hash value in the software.
>
> The other way is calculating the hash value on the send path, and choose the tx queue based on the send table, letting the host to decide which queue to send packet (since the send table is given by host).
>
> I may implement the both and see which one has better performance.
>
> Thanks,
> Wei
>
>
>
> -----Original Message-----
> From: adrian.chadd at gmail.com [mailto:adrian.chadd at gmail.com] On Behalf Of Adrian Chadd
> Sent: Tuesday, August 12, 2014 2:27 AM
> To: Wei Hu
> Cc: d at delphij.net; freebsd-net at freebsd.org
> Subject: Re: vRSS support on FreeBSD
>
> On 11 August 2014 02:48, Wei Hu <weh at microsoft.com> wrote:
>> CC freebsd-net@ for wider discussion.
>>
>> Hi Adrian,
>>
>> Many thanks for the explanation.  I checked the if_igb.c  and found the flowid field was set in the RX side in igb_rxeof():
>>
>> Igb_rxeof()
>> {
>>  ...
>> #ifdef  RSS
>>                         /* XXX set flowtype once this works right */
>>                         rxr->fmp->m_pkthdr.flowid =
>>                             le32toh(cur->wb.lower.hi_dword.rss);
>>                         rxr->fmp->m_flags |= M_FLOWID;  ...
>> }
>>
>> I have two questions regarding this.
>>
>> 1. Is the RSS hash value stored in cur->wb.lower.hi_dword.rss set by the NIC hardware?
>
> Yup.
>
>> 2. So the hash value and m_flags are stored in the mbuf related to the received packet on the rx side(lgb_rxeof()). But we check the hash value and m_flags in mbuf related to the send packet on the tx side (in igb_mq_start()). Does the kernel re-use the same mbuf for tx? If so, how does it know for the same network stream it should use the same mbuf got from the rx for packet sending? If not, how does the kernel preserve the same hash value across the rx mbuf and tx mbuf for same network stream? This seems quite magical to me.
>
> The mbuf flowid/flowtype ends up in the inpcb->inp_flowid /
> inpcb->inp_flowtype as part of the TCP receive path.
>
> Then whenever the TCP code outputs an mbuf, it copies the inpcb flow details out to outbound mbufs.
>
>>
>> For the Hyper-V case, the host controls which vCPU it wants to interrupt. And the rule can change dynamically based on the load. For a non-busy VM, host will send most packets to same vCPU for power saving purpose. For a busy VM, host will distribute the packets evenly across all vCPUs. This means host could change the RSS bucket mapping dynamically. Hyper-V does this by sending a mapping table to VM whenever the it needs update. This also means we cannot use FreeBSD's own bucket mapping which I believe is fixed. Also Hyper-V use its own hash key. So do you think it is possible we still use the exisiting RSS infrastructure built in FreeBSD in this purpose?
>
> Eventually. Doing rebalancing in RSS is on the TODO list, after I get the rest of the basic packet handling / routing done.
>
> How's vRSS notify the VM that the mapping table has changed? What's the format of it look like?
>
>
> -a