read() returns ETIMEDOUT on steady TCP connection

Andre Oppermann andre at freebsd.org
Fri May 2 19:57:39 UTC 2008


Mark Hills wrote:
> On Wed, 23 Apr 2008, Andre Oppermann wrote:
> 
>> http://people.freebsd.org/~andre/tcp_output-error-log.diff
>>
>> Please apply this patch and enable the sysctl net.inet.tcp.log_debug=1
>> and report any output.  You likely get some (normal) noise from syncache.
>> What we are looking for is reports from tcp_output.
> 
> Hi Andre, I've applied the patch and tested.
> 
> Aside from syncache noise, I get a constant stream of 'error 55' 
> (ENOBUFS?), once the number of connection gets to around 150 at 192kbps.
> 
> TCP: [192.168.5.43]:52153 to [192.168.5.40]:8080; tcp_output: error 55 
> while sending
> 
> 192.168.5.40 is the IP address of this host, running the server.
> 
> I tried to correlate the point of the application receiving ETIMEDOUT 
> with these messages, but that is tricky as it seems to be outputting a 
> lot of messages, and multiple messages over eachother (see below).
> 
> Because of the mention of no buffer space available, I checked the 
> values of net.inet.tcp.sendbuf* and recvbuf*, and increased the max 
> values with no effect.
> 
> When I get time I will modify the kernel to print errors which aren't 
> ENOBUFS to see if there are any others. But in the meantime, this sounds 
> like a problem to me. Is that correct?
> 
> Mark
> 
> 
> :8080; tcp_output: error 55 while sending
> TCP: [192.168.5.42]:57384T CtPo:  
> [[119922..116688..55..4402]]::85048400;1  ttoc p[_1o9u2t.p1u6t8:. 
> 5e.r4r0o]r: 8080;5 5t cwp_hoiultep uste:n deirnrgor 55 while sending
> TCP: [192.168.5.42]:57382 to [192.168.5.40]:8080; tcp_output: error 55 
> while sending
> TCP: [192.168.5.42]:57381 to [192.168.5.40]:8080; tcp_output: error 55 
> while sending
> TCP: [192.168.5.42]:57380 to [192.168.5.40]:8080; tcp_output: error 55 
> while sending

After tracing through the code it seems you are indeed memory limited.
Looking back at the netstat -m output:

  12550/250/12800/12800 4k (page size) jumbo clusters in use
  (current/cache/total/max)
  0/0/0 requests for jumbo clusters denied (4k/9k/16k)

This shows that the supply of 4k jumbo clusters is pretty much exhausted.
The cache may be allocated to different CPUs and the one making the request
at a given point may be depleted and can't get any from the global pool.
The big question is why the denied counter doesn't report anything.  I've
looked at the code paths and don't see any obvious reason why it doesn't
get counted.  Maybe Robert can give some insight here.

Try doubling the amount of 4k page size jumbo mbufs.  They are the primary
workhorse in the kernel right now:

  sysctl kern.ipc.nmbjumbop=25600

This should get further.  Still more may be necessary depending on workloads.

-- 
Andre



More information about the freebsd-net mailing list