Some performance measurements on the FreeBSD network stack

[Andre Oppermann]

On 20.04.2012 00:03, Luigi Rizzo wrote:
> On Thu, Apr 19, 2012 at 11:20:00PM +0200, Andre Oppermann wrote:
>> On 19.04.2012 22:46, Luigi Rizzo wrote:
>>> The allocation happens while the code has already an exclusive
>>> lock on so->snd_buf so a pool of fresh buffers could be attached
>>> there.
>>
>> Ah, there it is not necessary to hold the snd_buf lock while
>> doing the allocate+copyin.  With soreceive_stream() (which is
>
> it is not held in the tx path either -- but there is a short section
> before m_uiotombuf() which does
>
> 	...
> 	SOCKBUF_LOCK(&so->so_snd);
> 	// check for pending errors, sbspace, so_state
> 	SOCKBUF_UNLOCK(&so->so_snd);
> 	...
>
> (some of this is slightly dubious, but that's another story)

Indeed the lock isn't held across the m_uiotombuf().  You're talking
about filling an sockbuf mbuf cache while holding the lock?

>>> But the other consideration is that one could defer the mbuf allocation
>>> to a later time when the packet is actually built (or anyways
>>> right before the thread returns).
>>> What i envision (and this would fit nicely with netmap) is the following:
>>> - have a (possibly readonly) template for the headers (MAC+IP+UDP)
>>>    attached to the socket, built on demand, and cached and managed
>>>    with similar invalidation rules as used by fastforward;
>>
>> That would require to cross-pointer the rtentry and whatnot again.
>
> i was planning to keep a copy, not a reference. If the copy becomes
> temporarily stale, no big deal, as long as you can detect it reasonably
> quiclky -- routes are not guaranteed to be correct, anyways.

Be wary of disappearing interface pointers...

>>> - possibly extend the pru_send interface so one can pass down the uio
>>>    instead of the mbuf;
>>> - make an opportunistic buffer allocation in some place downstream,
>>>    where the code already has an x-lock on some resource (could be
>>>    the snd_buf, the interface, ...) so the allocation comes for free.
>>
>> ETOOCOMPLEXOVERTIME.
>
> maybe. But i want to investigate this.

I fail see what passing down the uio would gain you.  The snd_buf lock
isn't obtained again after the copyin.  Not that I want to prevent you
from investigating other ways. ;)

-- 
Andre