How to enable tcp bbr in FreeBSD???

Sun Apr 26 14:28:48 UTC 2020

Question for you…

I went hunting through these paths.. and found something interesting.

How does one specify in the socket API OOB send or recv?

I believe its MSG_OOB (and of course we also have the MSG_EOF flag too).

In sosend_generic(), sosend_stream() and sosend_dgram() we have a conversion that
happens … MSG_OOB -> PRUS_OOB <and> MSG_EOF -> PRUS_EOF.

But, as you point out here (which I had forgotten) TCP does *not* use
sosend_stream() or sosend_generic() anymore..

So how does the socket API MSG_OOB get translated into PRUS_OOB?

Its not in sosend() which calls the pru_send()

And its not in tcp_usr_send which is what the pru_send points to 
for TCP….

So how does this all occur?? I think MSG_OOB has been broken for
some time would be my guess… or am I missing something ???

R

> On Apr 26, 2020, at 9:25 AM, Michael Tuexen <tuexen at freebsd.org> wrote:
> 
> 
> 
>> On 26. Apr 2020, at 15:09, Randall Stewart <rrs at netflix.com> wrote:
>> 
>> I am thinking that this really needs to have
>> a deeper support in the transport.
>> 
>> I see that OOB is even in socket_dgram process and there
>> are comments in there that indicate it may be a problem.. I 
>> know UDP does not support it.
>> 
>> So I think what is needed here is
>> 
>> 1) a new pru_method that at send/rcv it can query if PRUS_EOF <or> PRUS_OOB
>>  is supported.
> Not sure what you need to check for PRUS_EOF?
>> 2) For UDP either query would return false.
>> 3) For TCP this would resolve to a stack specific query. For the 
>>  freebsd stack, it would return true for both, for Rack or BBR
>>  you would get True for PRUS_EOF and False for PRUS_OOB. This way
>>  you could capture an error in any case by adding a check at the
>>  top of send/recv and immediately return an error as appropriate.
>> 4) I would also think like all pru methods, you get a default of true/true
>>  so that way I guess unix domain sockets would continue as they are (not
>>  sure if they support these or not I should probably look)
> Hmm. Thinking about this:
> 
> I guess we want to focus on TCP, since my understanding is that the
> problem is that some TCP stacks do support OOB, some don't. So you
> can't query that right now via a TCP value.
> 
> So couldn't we check in tcp_usr_send() if the stack currently being
> used for the socket support OOB? It would be a stack specific value.
> Such a change should fix this issue and does not impact other protocols.
> 
> Best regards
> Michael
>> 
>> 
>> R
>> 
>>> On Apr 26, 2020, at 8:55 AM, Randall Stewart <rrs at netflix.com> wrote:
>>> 
>>> Sure..
>>> 
>>> I will take a look at it. 
>>> 
>>> R
>>> 
>>>> On Apr 26, 2020, at 8:51 AM, Michael Tuexen <Michael.Tuexen at macmic.franken.de> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On 26. Apr 2020, at 13:34, Randall Stewart via freebsd-transport <freebsd-transport at freebsd.org> wrote:
>>>>> 
>>>>> I have pulled down the reproducers.. one thing to note
>>>>> is they are all rack (though it could be that the problems
>>>>> are also in BBR). And of course FreeBSD is behind NF in
>>>>> rack at least.
>>>>> 
>>>>> I need to work on getting things updated.. one thing Michael, 
>>>>> both Rack and BBR in NF have lost the OOB handling. Please do
>>>>> not commit any more changes to Rack .. since that work has already
>>>>> been done.
>>>> I understand that the support of MSG_OOB is gone, but if you want
>>>> to return an error to the user when he uses MSG_OOB, you need to
>>>> trigger this error in the protocol specific code.
>>>> 
>>>> I don't think we can return an error in all cases (also for the
>>>> default stack), since that would change existing behaviour.
>>>> 
>>>> I leave this up to you.
>>>> 
>>>> Best regards
>>>> Michael
>>>>> 
>>>>> R
>>>>> 
>>>>>> On Apr 26, 2020, at 7:28 AM, Randall Stewart <rrs at netflix.com> wrote:
>>>>>> 
>>>>>> This is actually the first I have heard of these bugs…
>>>>>> 
>>>>>> I will have look at them Mark.
>>>>>> 
>>>>>> R
>>>>>> 
>>>>>>> On Apr 24, 2020, at 10:23 AM, Mark Johnston <markj at freebsd.org> wrote:
>>>>>>> 
>>>>>>> On Fri, Apr 24, 2020 at 03:15:08PM +0100, Tom Jones wrote:
>>>>>>>> rrs at freebsd.org
>>>>>>>> Bcc: 
>>>>>>>> Subject: Re: How to enable tcp bbr in FreeBSD???
>>>>>>>> Reply-To: 
>>>>>>>> In-Reply-To: <6042155a-297b-d85e-1d64-24d93da329a2 at gmail.com>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ... snip ...
>>>>>>>>> 
>>>>>>>>> Maybe it is not ready for prime time, i do not know why it is not in the
>>>>>>>>> default build.
>>>>>>>>> Maybe ask the committer.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> I have added rrs@ in cc and the freebsd-transport list. 
>>>>>>>> 
>>>>>>>> Does anyone know if there are plans to enable alternate TCP stacks in
>>>>>>>> generic? 
>>>>>>>> 
>>>>>>>> Is there a stability point we need to hit first?
>>>>>>> 
>>>>>>> There are a couple of open bugs found by syzkaller (complete with
>>>>>>> reproducers) that appeared when I enabled the alternate TCP stacks:
>>>>>>> 
>>>>>>> https://syzkaller.appspot.com/bug?id=986b4cecd84439df9794bda1a45d9cf0f50356fe
>>>>>>> https://syzkaller.appspot.com/bug?id=048f650e99696f881872a285cef0e3b9bd4f4e25
>>>>>>> 
>>>>>>> I'd expect these to be fixed before providing the alternate stacks in
>>>>>>> GENERIC.
>>>>>>> _______________________________________________
>>>>>>> freebsd-transport at freebsd.org mailing list
>>>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-transport
>>>>>>> To unsubscribe, send any mail to "freebsd-transport-unsubscribe at freebsd.org"
>>>>>> 
>>>>>> ------
>>>>>> Randall Stewart
>>>>>> rrs at netflix.com
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> ------
>>>>> Randall Stewart
>>>>> rrs at netflix.com
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> freebsd-transport at freebsd.org mailing list
>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-transport
>>>>> To unsubscribe, send any mail to "freebsd-transport-unsubscribe at freebsd.org"
>>> 
>>> ------
>>> Randall Stewart
>>> rrs at netflix.com
>>> 
>>> 
>>> 
>> 
>> ------
>> Randall Stewart
>> rrs at netflix.com

------
Randall Stewart
rrs at netflix.com