FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'

[Traiano Welcome]

Hi Mark


On 22/03/2012 13:54, "Mark Blackman" <mark at exonetric.com> wrote:

>
>On 22 Mar 2012, at 11:40, Traiano Welcome wrote:
>
>> That's what I thought as well, but it's the details that evade me.
>>Almost
>> all traffic to and from this server is UDP (syslog), the graph I sent
>> earlier shows the kind of volumes and trends that are typical: Peak
>> traffic during the problem periods averages at about 1 Mbps outbound and
>> 200 Kbps inbound to/from the interface. The interface itself is a
>> Embedded Broadcom 5708 NIC on a Dell PowerEdge 1950.
>> 
>> 
>> Here are a couple of netstat polls during one of the problem periods:
>> 
>> ----
>> [root at syslog2]# date;netstat -p udp -s |egrep -w
>> "(received|delivered|dropped)"
>> Thu Mar 22 12:11:34 SAST 2012
>>       19969 datagrams received
>>       2 dropped due to no socket
>>       0 dropped due to full socket buffers
>>       19967 delivered
>> .
>> .
>> .
>> [root at syslog2~]# date;netstat -p udp -s |egrep -w
>> "(received|delivered|dropped)"
>> Thu Mar 22 13:36:46 SAST 2012
>>       662385 datagrams received
>>       118 dropped due to no socket
>>       0 dropped due to full socket buffers
>>       662267 delivered
>> ---
>> 
>> 
>> Somehow this doesn't strike  me as a large volume of throughput Š
>
>Ok, fair enough. You might try simulating the problem by deliberately
>overloading the syslog UDP output and confirm the cause.


Apparently this means that the network driver has "filled" up with
packets. John Baldwin over at freebsd-net@ advises I up the number of
descriptors assigned to igb to the maximum
to workaround this using the hw.igb.maxtxd tunable you would set. So I've
rebooted with the following in loader.conf:

hw.igb.rxd=4096
hw.igb.txd=4096


This seems to be working so far. What I've noticed is that the system is
using far  less RAM than previously, and CPU utilisation is up to 100% of
one core, load average is 1, which I would guess means that the system is
now processing a lot more syslog  data now that "more packets are making
it through the network driver".

I'll keep monitoring over a 24 hour period though, to see how effective
this is.



>
>- Mark