read() returns ETIMEDOUT on steady TCP connection

[Andre Oppermann]

Deng XueFeng wrote:
> hi,
> the patch can not apply to 6.2, cound do a new patch for 6.2 or 6.3 ?

The logging function is not (yet) present in RELENG_6.  I'll post the
patch when I've backported the functionality.

However it's an important information that it happens on 6.2 too.  That
means the source of the trouble wasn't introduced with 7.0.

-- 
Andre

>> I've looked at the code paths again.  There are two possibilities:
>>
>>   a) the mbuf allocator has some anomaly where it rejects memory requests
>>      but doesn't update the statistics (the code is there however).
>>
>>   b) the error doesn't come from the mbuf allocation but from ip_output()
>>      and further down the chain.
>>
>> To differentiate please try this updated patch and report the log output
>> again (don't forget to set net.inet.tcp.log_debug=1):
>>
>>   http://people.freebsd.org/~andre/tcp_output-error-log.diff
>>
>> -- 
>> Andre
>>
>> Deng XueFeng wrote:
>>> hi
>>> I'am also meet this problem in my mss server(missey streaming server).
>>> one encoder push stream to mss, then run 100 client player playing the
>>> sream, as the client number increase,  mss  will occur this error sooner or later
>>> like this:
>>> I'am using kqueue, and will got a event with  EV_EOF and fflags =
>>> ETIMEDOUT,
>>> if i ignore EV_EOF  flag, then ETIMEDOUT will be return by read(2),
>>>
>>> and the tcpdump also show that server  will send RST packet to encoder.
>>>
>>>
>>>> Hello,
>>>>
>>>> I'm are having a trouble with TCP connections being dropped with "read: 
>>>> Operation timed out". What is unusual is that this is happening right in 
>>>> the middle of sending a steady stream of data with no network congestion.
>>>>
>>>> The system is FreeBSD 7 and a bespoke streaming server with 1Gbit 
>>>> connection. The server receives a 192kbps inbound stream over TCP, and 
>>>> broadcasts it over a large number of TCP streams.
>>>>
>>>> With no visible or obvious pattern, the inbound read() fails with 
>>>> ETIMEDOUT. The likelihood of this happening seems to increase as the 
>>>> number of audience connections increases. It's happens every few minutes 
>>>> even with a small audience (eg. 300 outbound connections and about 
>>>> 60mbit).
>>>>
>>>> It doesn't cough and splutter -- steady data is coming in, then it just 
>>>> drops the connection.
>>>>
>>>> systat doesn't show problems inbound; all packets received are delivered 
>>>> to the upper layer. But on outbound, there is consistent 'output drops':
>>>>
>>>>      IP Output
>>>> 7028 total packets sent
>>>> 7028 - generated locally
>>>>   314 - output drops
>>>>
>>>> As the number of outbound connections increases, the 'output drops' 
>>>> increases to around 10% of the total packets sent and maintains that 
>>>> ratio. There's no problems with network capacity.
>>>>
>>>> I've tried different servers, different network interfaces (bge, em), 
>>>> different kernel (7-RELEASE, 7-STABLE). Have also checked dev.bge.0.stats 
>>>> and dev.em.0.stats for CRC errors etc. which show no problems. 'netstat 
>>>> -m' doesn't show any reaching of mbuf and sbuf limits. The problem is seen 
>>>> in a dedicated, uncontended test environment.
>>>>
>>>> Can anyone explain why the packets are being dropped outbound, and how 
>>>> this could affect inbound TCP data in such an abrupt way? What can I do to 
>>>> solve this?
>>>>
>>>> Thanks,
>>>>
>>>> Mark
>>>> _______________________________________________
>>>> freebsd-net at freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>