syncache errors

[Andre Oppermann]

On 29.08.2010 19:16, Mike Tancsa wrote:
> At 11:33 AM 8/29/2010, Andre Oppermann wrote:
>> On 29.08.2010 16:09, Mike Tancsa wrote:
>>> After upgrading to a recent STABLE, I have been seeing the following sporadic errors
>>>
>>> Aug 28 04:15:15 smarthost2 kernel: TCP: [xx.yy.165.120]:53617 to [xx.yy.164.50]:25; syncache_socket:
>>> Socket create failed due to limits or memory shortage
>>>
>>> this is with
>>> FreeBSD 8.1-STABLE #7: Wed Aug 25 15:32:05 EDT 2010
>>> and the previous kernel was from July 20th.
>>>
>>> The odd thing is that the error is triggered from a RELENG_6 host only it would seem. I noticed
>>> there are a number of syncache and tcp updates to RELENG_8, is it possible this is related ? I dont
>>> have any specific tweaks in /etc/sysctl.conf
>>>
>>> other than
>>> net.inet.tcp.log_in_vain: 1
>>> net.inet.udp.log_in_vain: 1
>>
>> The log_in_vain sysctl would cause the logging of syncache errors
>> as well (net.inet.tcp.log_debug). This was a POLA violation and
>> I've separated them on August 27 on 8-stable. So if you update
>> you won't see them anymore. That doesn't change the fact that
>> the socket create failed though.
>>
>> If it only happens from a RELENG_6 box it probably is a problem
>> with port re-usage by RELENG_6. The difficulty is that sonewconn()
>> fails and doesn't return an error code. Hence it may be one of
>> listen queue limits reached, memory shortage or a problem with
>> inserting the inpcb into the hash lists.
>
> Actually, I think I might have found the issue. I was focusing on the "memory" part not the limits.
> The 'RELENG_6 only' is just a fluke as thats a box that sends a lot of email to this particular
> server. It turns out, sendmail was rate limiting the server
> sm-mta[25923]: deferring connections on daemon IPv4: 8 per second
> and somehow syncache is aware/logs this now where as it did not before ?

When sendmail is deferring the connections it should not show up
in the syncache logs.  The accept() by sendmail is much later than
when the socket for a new connection is created in syncache.  Here
it points to the limits in the listen queue.  Maybe sendmail is
getting behind in accepting new connections and the listen queue
is overflowing.

-- 
Andre