How to enable tcp bbr in FreeBSD???

Randall Stewart rrs at netflix.com
Sun Apr 26 15:32:45 UTC 2020


By the way

I did run the reproducers on both the rack and bbr stack
that is in the phabricator.. and no crash.

So even without the OOB changes the crash is gone with the
latest update.. though we do need to get the USER API fixed
here so that when OOB is not supported an error is returned…

R

> On Apr 26, 2020, at 11:18 AM, Randall Stewart <rrs at netflix.com> wrote:
> 
> I will also follow up after I get these in with changes
> that will expand the OOB check by adding the appropriate tcp stack level
> query and of course BBR and Rack will say no to PRUS_OOB (ENOTSUPPORTED)
> 
> R
> 
>> On Apr 26, 2020, at 11:17 AM, Randall Stewart <rrs at netflix.com> wrote:
>> 
>> Michael:
>> 
>> So
>> 
>> https://reviews.freebsd.org/D24574
>> 
>> Is a first step for OOB handling on a per stack basis.
>> First question I have is does OOB even work, I suspect not.
>> 
>> Second here is a prep patch for getting the latest rack and bbr in …
>> 
>> https://reviews.freebsd.org/D24575
>> 
>> And then finally on top of that is the update to the latest rack and bbr
>> stack:
>> 
>> https://reviews.freebsd.org/D24576
>> 
>> Now there is something odd about 24576 it says there are 7k changes to rack (which
>> is right) but I don’t see them in the review.. not sure if Phabricator is having
>> difficulty or what..
>> 
>> The patch file I submitted definitely had them .. hmm
>> 
>> R
>> 
>> 
>> 
>>> On Apr 26, 2020, at 9:25 AM, Michael Tuexen <tuexen at freebsd.org> wrote:
>>> 
>>> 
>>> 
>>>> On 26. Apr 2020, at 15:09, Randall Stewart <rrs at netflix.com> wrote:
>>>> 
>>>> I am thinking that this really needs to have
>>>> a deeper support in the transport.
>>>> 
>>>> I see that OOB is even in socket_dgram process and there
>>>> are comments in there that indicate it may be a problem.. I 
>>>> know UDP does not support it.
>>>> 
>>>> So I think what is needed here is
>>>> 
>>>> 1) a new pru_method that at send/rcv it can query if PRUS_EOF <or> PRUS_OOB
>>>> is supported.
>>> Not sure what you need to check for PRUS_EOF?
>>>> 2) For UDP either query would return false.
>>>> 3) For TCP this would resolve to a stack specific query. For the 
>>>> freebsd stack, it would return true for both, for Rack or BBR
>>>> you would get True for PRUS_EOF and False for PRUS_OOB. This way
>>>> you could capture an error in any case by adding a check at the
>>>> top of send/recv and immediately return an error as appropriate.
>>>> 4) I would also think like all pru methods, you get a default of true/true
>>>> so that way I guess unix domain sockets would continue as they are (not
>>>> sure if they support these or not I should probably look)
>>> Hmm. Thinking about this:
>>> 
>>> I guess we want to focus on TCP, since my understanding is that the
>>> problem is that some TCP stacks do support OOB, some don't. So you
>>> can't query that right now via a TCP value.
>>> 
>>> So couldn't we check in tcp_usr_send() if the stack currently being
>>> used for the socket support OOB? It would be a stack specific value.
>>> Such a change should fix this issue and does not impact other protocols.
>>> 
>>> Best regards
>>> Michael
>>>> 
>>>> 
>>>> R
>>>> 
>>>>> On Apr 26, 2020, at 8:55 AM, Randall Stewart <rrs at netflix.com> wrote:
>>>>> 
>>>>> Sure..
>>>>> 
>>>>> I will take a look at it. 
>>>>> 
>>>>> R
>>>>> 
>>>>>> On Apr 26, 2020, at 8:51 AM, Michael Tuexen <Michael.Tuexen at macmic.franken.de> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On 26. Apr 2020, at 13:34, Randall Stewart via freebsd-transport <freebsd-transport at freebsd.org> wrote:
>>>>>>> 
>>>>>>> I have pulled down the reproducers.. one thing to note
>>>>>>> is they are all rack (though it could be that the problems
>>>>>>> are also in BBR). And of course FreeBSD is behind NF in
>>>>>>> rack at least.
>>>>>>> 
>>>>>>> I need to work on getting things updated.. one thing Michael, 
>>>>>>> both Rack and BBR in NF have lost the OOB handling. Please do
>>>>>>> not commit any more changes to Rack .. since that work has already
>>>>>>> been done.
>>>>>> I understand that the support of MSG_OOB is gone, but if you want
>>>>>> to return an error to the user when he uses MSG_OOB, you need to
>>>>>> trigger this error in the protocol specific code.
>>>>>> 
>>>>>> I don't think we can return an error in all cases (also for the
>>>>>> default stack), since that would change existing behaviour.
>>>>>> 
>>>>>> I leave this up to you.
>>>>>> 
>>>>>> Best regards
>>>>>> Michael
>>>>>>> 
>>>>>>> R
>>>>>>> 
>>>>>>>> On Apr 26, 2020, at 7:28 AM, Randall Stewart <rrs at netflix.com> wrote:
>>>>>>>> 
>>>>>>>> This is actually the first I have heard of these bugs…
>>>>>>>> 
>>>>>>>> I will have look at them Mark.
>>>>>>>> 
>>>>>>>> R
>>>>>>>> 
>>>>>>>>> On Apr 24, 2020, at 10:23 AM, Mark Johnston <markj at freebsd.org> wrote:
>>>>>>>>> 
>>>>>>>>> On Fri, Apr 24, 2020 at 03:15:08PM +0100, Tom Jones wrote:
>>>>>>>>>> rrs at freebsd.org
>>>>>>>>>> Bcc: 
>>>>>>>>>> Subject: Re: How to enable tcp bbr in FreeBSD???
>>>>>>>>>> Reply-To: 
>>>>>>>>>> In-Reply-To: <6042155a-297b-d85e-1d64-24d93da329a2 at gmail.com>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> ... snip ...
>>>>>>>>>>> 
>>>>>>>>>>> Maybe it is not ready for prime time, i do not know why it is not in the
>>>>>>>>>>> default build.
>>>>>>>>>>> Maybe ask the committer.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I have added rrs@ in cc and the freebsd-transport list. 
>>>>>>>>>> 
>>>>>>>>>> Does anyone know if there are plans to enable alternate TCP stacks in
>>>>>>>>>> generic? 
>>>>>>>>>> 
>>>>>>>>>> Is there a stability point we need to hit first?
>>>>>>>>> 
>>>>>>>>> There are a couple of open bugs found by syzkaller (complete with
>>>>>>>>> reproducers) that appeared when I enabled the alternate TCP stacks:
>>>>>>>>> 
>>>>>>>>> https://syzkaller.appspot.com/bug?id=986b4cecd84439df9794bda1a45d9cf0f50356fe
>>>>>>>>> https://syzkaller.appspot.com/bug?id=048f650e99696f881872a285cef0e3b9bd4f4e25
>>>>>>>>> 
>>>>>>>>> I'd expect these to be fixed before providing the alternate stacks in
>>>>>>>>> GENERIC.
>>>>>>>>> _______________________________________________
>>>>>>>>> freebsd-transport at freebsd.org mailing list
>>>>>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-transport
>>>>>>>>> To unsubscribe, send any mail to "freebsd-transport-unsubscribe at freebsd.org"
>>>>>>>> 
>>>>>>>> ------
>>>>>>>> Randall Stewart
>>>>>>>> rrs at netflix.com
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> ------
>>>>>>> Randall Stewart
>>>>>>> rrs at netflix.com
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> freebsd-transport at freebsd.org mailing list
>>>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-transport
>>>>>>> To unsubscribe, send any mail to "freebsd-transport-unsubscribe at freebsd.org"
>>>>> 
>>>>> ------
>>>>> Randall Stewart
>>>>> rrs at netflix.com
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> ------
>>>> Randall Stewart
>>>> rrs at netflix.com
>> 
>> ------
>> Randall Stewart
>> rrs at netflix.com
>> 
>> 
>> 
> 
> ------
> Randall Stewart
> rrs at netflix.com
> 
> 
> 

------
Randall Stewart
rrs at netflix.com





More information about the freebsd-transport mailing list