How to enable tcp bbr in FreeBSD???
Michael Tuexen
tuexen at freebsd.org
Sun Apr 26 16:53:53 UTC 2020
> =On 26. Apr 2020, at 16:28, Randall Stewart <rrs at netflix.com> wrote:
>
> Question for you…
>
> I went hunting through these paths.. and found something interesting.
>
> How does one specify in the socket API OOB send or recv?
>
> I believe its MSG_OOB (and of course we also have the MSG_EOF flag too).
Correct.
>
> In sosend_generic(), sosend_stream() and sosend_dgram() we have a conversion that
> happens … MSG_OOB -> PRUS_OOB <and> MSG_EOF -> PRUS_EOF.
OK. Interesting to note here is that
MSG_OOB == 0x0001
PRUS_OOB == 0x0001
MSG_EOF == 0x0100
PRUS_EOF == 0x0002
So the conversion of MSG_OOB -> PRUS_OOB is a NOOP, whereas MSG_EOF -> PRUS_EOF is not.
>
> But, as you point out here (which I had forgotten) TCP does *not* use
> sosend_stream() or sosend_generic() anymore..
>
> So how does the socket API MSG_OOB get translated into PRUS_OOB?
Let me have a look.
>
> Its not in sosend() which calls the pru_send()
>
> And its not in tcp_usr_send which is what the pru_send points to
> for TCP….
>
> So how does this all occur?? I think MSG_OOB has been broken for
> some time would be my guess… or am I missing something ???
It can be. I have NEVER used it. I think the urgent bit does not
work well between stacks. It just happened that syzkaller likes
MSG_OOB and paniced the stack. While looking at the RACK stack
I figured out that RACK is not intended to work with MSG_OOB.
That is why I suggested as a fix to return an error to the user.
That does not mean that I say MSG_OOB works for the base stack.
It just doesn't panic. Richard found other strange things with
MSG_OOB with the base stack. SO it doesn't panic, but it might
not work at all. Will need to do some basic testing. I just added
support for the urgent bit to the FreeBSD version of packetdrill
last Wednesday...
Best regards
Michael
>
> R
>
>
>> On Apr 26, 2020, at 9:25 AM, Michael Tuexen <tuexen at freebsd.org> wrote:
>>
>>
>>
>>> On 26. Apr 2020, at 15:09, Randall Stewart <rrs at netflix.com> wrote:
>>>
>>> I am thinking that this really needs to have
>>> a deeper support in the transport.
>>>
>>> I see that OOB is even in socket_dgram process and there
>>> are comments in there that indicate it may be a problem.. I
>>> know UDP does not support it.
>>>
>>> So I think what is needed here is
>>>
>>> 1) a new pru_method that at send/rcv it can query if PRUS_EOF <or> PRUS_OOB
>>> is supported.
>> Not sure what you need to check for PRUS_EOF?
>>> 2) For UDP either query would return false.
>>> 3) For TCP this would resolve to a stack specific query. For the
>>> freebsd stack, it would return true for both, for Rack or BBR
>>> you would get True for PRUS_EOF and False for PRUS_OOB. This way
>>> you could capture an error in any case by adding a check at the
>>> top of send/recv and immediately return an error as appropriate.
>>> 4) I would also think like all pru methods, you get a default of true/true
>>> so that way I guess unix domain sockets would continue as they are (not
>>> sure if they support these or not I should probably look)
>> Hmm. Thinking about this:
>>
>> I guess we want to focus on TCP, since my understanding is that the
>> problem is that some TCP stacks do support OOB, some don't. So you
>> can't query that right now via a TCP value.
>>
>> So couldn't we check in tcp_usr_send() if the stack currently being
>> used for the socket support OOB? It would be a stack specific value.
>> Such a change should fix this issue and does not impact other protocols.
>>
>> Best regards
>> Michael
>>>
>>>
>>> R
>>>
>>>> On Apr 26, 2020, at 8:55 AM, Randall Stewart <rrs at netflix.com> wrote:
>>>>
>>>> Sure..
>>>>
>>>> I will take a look at it.
>>>>
>>>> R
>>>>
>>>>> On Apr 26, 2020, at 8:51 AM, Michael Tuexen <Michael.Tuexen at macmic.franken.de> wrote:
>>>>>
>>>>>
>>>>>
>>>>>> On 26. Apr 2020, at 13:34, Randall Stewart via freebsd-transport <freebsd-transport at freebsd.org> wrote:
>>>>>>
>>>>>> I have pulled down the reproducers.. one thing to note
>>>>>> is they are all rack (though it could be that the problems
>>>>>> are also in BBR). And of course FreeBSD is behind NF in
>>>>>> rack at least.
>>>>>>
>>>>>> I need to work on getting things updated.. one thing Michael,
>>>>>> both Rack and BBR in NF have lost the OOB handling. Please do
>>>>>> not commit any more changes to Rack .. since that work has already
>>>>>> been done.
>>>>> I understand that the support of MSG_OOB is gone, but if you want
>>>>> to return an error to the user when he uses MSG_OOB, you need to
>>>>> trigger this error in the protocol specific code.
>>>>>
>>>>> I don't think we can return an error in all cases (also for the
>>>>> default stack), since that would change existing behaviour.
>>>>>
>>>>> I leave this up to you.
>>>>>
>>>>> Best regards
>>>>> Michael
>>>>>>
>>>>>> R
>>>>>>
>>>>>>> On Apr 26, 2020, at 7:28 AM, Randall Stewart <rrs at netflix.com> wrote:
>>>>>>>
>>>>>>> This is actually the first I have heard of these bugs…
>>>>>>>
>>>>>>> I will have look at them Mark.
>>>>>>>
>>>>>>> R
>>>>>>>
>>>>>>>> On Apr 24, 2020, at 10:23 AM, Mark Johnston <markj at freebsd.org> wrote:
>>>>>>>>
>>>>>>>> On Fri, Apr 24, 2020 at 03:15:08PM +0100, Tom Jones wrote:
>>>>>>>>> rrs at freebsd.org
>>>>>>>>> Bcc:
>>>>>>>>> Subject: Re: How to enable tcp bbr in FreeBSD???
>>>>>>>>> Reply-To:
>>>>>>>>> In-Reply-To: <6042155a-297b-d85e-1d64-24d93da329a2 at gmail.com>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ... snip ...
>>>>>>>>>>
>>>>>>>>>> Maybe it is not ready for prime time, i do not know why it is not in the
>>>>>>>>>> default build.
>>>>>>>>>> Maybe ask the committer.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have added rrs@ in cc and the freebsd-transport list.
>>>>>>>>>
>>>>>>>>> Does anyone know if there are plans to enable alternate TCP stacks in
>>>>>>>>> generic?
>>>>>>>>>
>>>>>>>>> Is there a stability point we need to hit first?
>>>>>>>>
>>>>>>>> There are a couple of open bugs found by syzkaller (complete with
>>>>>>>> reproducers) that appeared when I enabled the alternate TCP stacks:
>>>>>>>>
>>>>>>>> https://syzkaller.appspot.com/bug?id=986b4cecd84439df9794bda1a45d9cf0f50356fe
>>>>>>>> https://syzkaller.appspot.com/bug?id=048f650e99696f881872a285cef0e3b9bd4f4e25
>>>>>>>>
>>>>>>>> I'd expect these to be fixed before providing the alternate stacks in
>>>>>>>> GENERIC.
>>>>>>>> _______________________________________________
>>>>>>>> freebsd-transport at freebsd.org mailing list
>>>>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-transport
>>>>>>>> To unsubscribe, send any mail to "freebsd-transport-unsubscribe at freebsd.org"
>>>>>>>
>>>>>>> ------
>>>>>>> Randall Stewart
>>>>>>> rrs at netflix.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ------
>>>>>> Randall Stewart
>>>>>> rrs at netflix.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> freebsd-transport at freebsd.org mailing list
>>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-transport
>>>>>> To unsubscribe, send any mail to "freebsd-transport-unsubscribe at freebsd.org"
>>>>
>>>> ------
>>>> Randall Stewart
>>>> rrs at netflix.com
>>>>
>>>>
>>>>
>>>
>>> ------
>>> Randall Stewart
>>> rrs at netflix.com
>
> ------
> Randall Stewart
> rrs at netflix.com
>
>
>
More information about the freebsd-transport
mailing list