SCTP using questions (API etc.)

[Vadim Goncharov]

Hi Michael Tuexen! 

On Thu, 6 Mar 2008 09:34:13 +0100; Michael Tuexen wrote about 'Re: SCTP using questions (API etc.)':

>>>> "substreams". SCTP can do it for me, it's wonderful, but in practice
>>>> there
>>>> are some questions.
>>>>
>>>> How long can be one particular SCTP message? Can I rely on the fact
>>>> that it
>>>> can be unbounded, e.g. I want to emulate a stream with transfer of
>>>> 6 Gig-sized file?
>>> Protocol wise there is no limitation of the message size. API wise,  
>>> for
>>> this size of a message you need to use the explicit EOR mode to be  
>>> able
>>> to pass this large message using multiple sequential send() calls.
>>
>> And how should I determine from my/remote stack an optimal size for  
>> message
>> parts when entire message is guaranteed to not fit into buffers/ 
>> windows of
>> both peers?
> If the sendbuffer is too small for the message to fit, the send call
> will return -1 and errno being set to EMSGSIZE. Or you do it in the  
> application
> by inspecting the sendbuffer size. You do not have to deal with the  
> recv buffer
> of the peer.

So this means I need no subscription to unsent messages and simply can try
to resend message in several steps without EOR, after getting EMSGSIZE ?

>>>> Can a message be of zero-length data (only headers) ?
>>> Empty user messages, i.e. a DATA chunk without payload is not  
>>> allowed.
>>> An empty SCTP message, i.e. only the common header without any chunks
>>> is allowed and processed by FreeBSD when received, but ever send  
>>> (well,
>>> I do not know a way to force the FreeBSD implementation to send it).
>>
>> OK, understood. So I should include at least 1 byte of my own  
>> headers into
>> data and do receive into *iov with at least to parts to ensure good  
>> align
>> for non-header part?
> What header are you talking about? An application header or any SCTP  
> header?
> You will never receive any SCTP header as part of a user message via
> a recv() call. SCTP will give you as much of a message that fits into
> the buffer you provide or it has, if the partial delivery API has been  
> invoked.

My applicaion-protocol header, of course. Does this mean also that I should
always enable partial delivery on receiving? Or what will happen if received
msg is too big and don't fir into my buffers?

>>>> What is the relation between SCTP streams in both directions? Can
>>>> streams
>>>> be opened and closed on-demand, like SSH port forwarding (yet again
>>>> multiplexing example) or they are preallocated at connection setup  
>>>> all
>>>> together? What is the minimum number of streams application can rely
>>>> upon
>>>> (or it just one stream 0 in general case) ?
>>> If you restrict to protocols being in RFC status, there is no way of
>>> modifying the number of streams during the lifetime of an  
>>> association.
>>> The number of streams in each direction is negotiated during the
>>> association setup. The streams in bother directions are completely
>>> independent. There is always at least one stream in each direction,
>>> which
>>> is stream 0.
>>> However, there is an extension (currently specified in an Internet
>>> Draft)
>>> which allows the addition of streams during the lifetime of an
>>> association.
>>> The ID is at least partially supported by the FreeBSD implementation.
>>> https://datatracker.ietf.org/drafts/draft-stewart-sctpstrrst/
>>
>> OK. Are there recommended defaults for various stacks about how many
>> streams they are creating by default / what maximum of them  
>> application
>> can ever request?
> The maximum number to request is 2^16 - 1. It is controllable by the
> applications via socket options. Defaults in OSes are in the order of
> 10, 16, 32...

Can I be sure that every OS can give me maximum number of streams if I
request it?

>>>> How can I put request to kernel for a connect, for example, and then
>>>> sleep
>>>> until connect will complete or event in some another descriptor will
>>>> occur?
>>> If you use the 1-to-1 style API, it should be similar to using TCP.
>>> Put the socket in non-blocking mode, enable notifications,
>>> call connect() and wait until the fd becomes readable. You should  
>>> get an
>>> indication that that association has been established or could not
>>> start.
>>
>> Yes, that's possible, as I see after reading draft-ietf-tsvwg- 
>> sctpsocket.
>> But several more questions arise. What notifications do I really need
>> on 1-to-1 non-blocking socket API mode? What use is 'context' in this
>> 1-to-1 context and why after a failed send I must receive entire  
>> failed
>> sent message (which can be very long) instead of just an error code?
> The context is something you provide in the send call and is given
> back to you. So you can use it to find some state/buffer/whatever again.

It was unclear from draft whether context is one per SCTP association or per
send call. And what the hell are all that unsent messages, why I must
retrieve entire unsent message - can I fire-and-forget a 2M msg and receive
only context of it instead of all 2 megs? And on which condition such event
can ever occur - with TCP it's simple, I either do write() a number of bytes
successfully or receive an error from write() - be that EAGAIN for just
blocking of peer's recv() or connection termination error. What concept is
under unsent msgs?

>> In usual FSM I can use kqueue()/kevent() with arbitrary void* to my
>> data, also telling me how many bytes I can read from or write to
>> the socket (RCVLOWAT etc.), as well as indicating error/EOF conditions
>> so I don't need to do additional syscalls. Is this working with SCTP?
> Haven't tried it... Sounds like it would make sense to make sure that
> it works.

Oh, can you please check it?.. Would be good to support all features
described in kqueue(2).

>> If I can't write to TCP socket (due to window shortage from peer),
>> I leave data in my own application buffers, but SCTP tells something
>> about unsent messages delivered later, looks somewhat weird, do I  
>> really
>> need this? Also, all that msg*/cmsg* staff is too complex, and I see
>> there are simplier sctp_send()/sctp_sendx() interfaces, will they be
>> enough and really simplier for me?..
> sctp_sendx() purpose is to use the multiple addresses provided during
> the implicit setup of the association. So I think you are not looking  
> for

Ok.

> this. sctp_send() can be used to provide the stream id, payload protocol
> identifier and to on with using the CMSG stuff. So you might be looking
> for this function.

With CMSG? May be you wanted to say 'without' ?..

>>>> How can I put each client to it's fd and then do a kqueue()/kevent()
>>>> on a
>>>> set of those fd's (and other items) ? It is very handy to have this
>>>> architecture as kevent() allows to store an arbitrary void* in it's
>>>> structure which I can later use to quickly dispatch events.
>>>>
>>>> And, of course, all this usual C10K-problem-solving-TCP-server
>>>> tricks I want
>>>> with basic SCTP SEQPACKET benefits: multiple streams and message
>>>> record
>>>> separation (I don't need other SCTP features currently). Where can I
>>>> find
>>>> answers to these questions, like it was with W.R.Stevens books for
>>>> TCP ?..
>>> Have you looked at the third edition of 'Unix Network Programming'?
>>> Randall Stewart wrote a couple of sections covering SCTP...
>>
>> Unfortunately, I have only 2nd edition currently available here,  
>> though
>> heard about 3rd, yes. May be several months later...
> It is really worth buying if you are interested in SCTP socket  
> programming...

I know, but in my province it is currently unavailable for some months...
you know, Siberia, bears walking on the streets :) but it is not critical
for actual SCTP programming (TCP version will be debugged first), but I need
to take architectural decisions now.

Also, are there some examples of real-world SCTP applications with source
code available? May be something is getting to integrate into our base
system?..

-- 
WBR, Vadim Goncharov. ICQ#166852181       mailto:vadim_nuclight at mail.ru
[Moderator of RU.ANTI-ECOLOGY][FreeBSD][http://antigreen.org][LJ:/nuclight]