SCTP problem, how to debug?

[Michael Tuexen]

> On 17. Jul 2020, at 18:07, Bernd Walter <ticso at cicely7.cicely.de> wrote:
> 
> I'm running an LED matrix with SCTP.
> The matrix consists from 24 raspberry pi running NFS-root FreeBSD
> 12.0-RELEASE (they have an SD card for u-boot and loader).
> A client system is running FreeBSD 12.1-RELEASE.
I fixed iterator related bugs and this was MFCed to stable/12 recently.
The problem was that the iterator stopped sending.
> 
> The matrix modules have a one to many service socket.
> The daemon regularily sends status informations (temperature, etc.) to
> each association and has a second thread to receive.
> 
> The client system has two deamons running.
> 
> One daemon is to control power output related to temperature states.
> It has one thread reestablishing associations via sctp_connectx() to
> each of the 24 modules using a single one to many socket.
> Another thread collects all regular received data and updates stored
> values.
> Yet another thread sends power control data via SCTP_SENDALL, so that all modules know
> the maxed allowed brightness rating.
> 
> The other daemon uses the same threads to reconnect and receive.
> It connects to the very same sockets on the modules.
> Another thread updates picture data and wanted power rating.
> That is sending picture data to the given matrix module and then
> SCTP_SENDALL an update trigger to all modules.
> That is reduced brightness at night times, ...
> 
> All SCTP_SENDALL are just trigger with 0 length and different ppid values.
Are you really sending messages of length 0? That shouldn't work...
> 
> From time to time (1-5 days) I notice that a module won't get updates
> anymore.
> I see that the association got a SCTP_SENDER_DRY_EVENT event.
> Therefor my expectation is that there is nothing to send.
> I still see the association in the socket list and also receive the regular
> temperature data.
> However, obviously sending won't happen.
> The other modules still get data.
> 
> When I restart the client daemon, things start to work again.
> 
> Currently I'm clueless on how to debug this problem any fsurther.
Can you try stable/12?
> Doing a full network log would be too big and I'm not very experienced
> to understand the SCTP packets.
> I have no idea to see what data is in the send buffer.
> netstat with TCP would show send and receive queue, not so with SCTP.
> Data is send with a single thread, which sctp_sendmsgx() the data for
> all modules sequencially into the same socket.
I'm not sure I understood what you are actually doing on which socket
and how many associations are involved.

Best regards
Michael
> I havn't checked yet if I get an error with the write to the specific
> module IP.
> 
> -- 
> B.Walter <bernd at bwct.de> http://www.bwct.de
> Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"