nagios and freebsd threads issue : help please ...

Christophe Yayon lists at nbux.com
Fri Aug 19 18:33:43 GMT 2005


Hi all

You should know about freebsd and nagios 2.0b threads issues (100% cpu 
use by a forked process, lost check result, some pause of nagios main 
process in certains obscursives conditions...).

Some Nagios developpers says that the problem is in FreeBSD and some 
other says that the problem is in nagios pthreads implementation, here a 
resume of our discussions :

-------
The thread I started is here:

  http://marc.theaimsgroup.com/?t=111930118000001&r=1&w=2

  There are some very interesting replies, a few in particular note that
  Nagios may be breaking POSIX spec in how it spawns/destroys threads:

  http://marc.theaimsgroup.com/?l=freebsd-hackers&m=111944526323754&w=2
  http://marc.theaimsgroup.com/?l=freebsd-hackers&m=111945035012258&w=2

  Anyhow, I"m sure if Ethan were to post some more specific info to
  freebsd-hackers at fr... (it"s an open list, no need to sub), this
  issue could get banged out pretty quickly.

  Shortly after this thread, I found another where the issue was brought up
  by another curious poster, and he was using 5.4, which uses a newer
  threading library:

  http://marc.theaimsgroup.com/?t=112119712600002&r=1&w=2

  This post again brings up the "fork without exec or exit" possibly not
  following spec:

  http://marc.theaimsgroup.com/?l=freebsd-hackers&m=112125883804481&w=2

  "I don"t know what Nagios does just after fork(2), it would be worth to
  check.  It appears that fork(2)ing without exec(2)ing or _exit(2)ing
  in a pthreaded program is not a "valid" behaviour, regarding to
  SUSv3 [1].  I don"t want to avoid admitting there is a problem in
  FreeBSD threading library, I don"t know how other OSes handle this,
  but Nagios folks should really avoid doing what is explicitely
  dissuaded in SUSv3."
--------


--------
As the problem isn't in Nagios and noone seems to have an authoritative
  answer on what exactly is causing it, I'd say you would be better off
  switching to a GNU/Linux system, with at least Linux 2.4.29 and
  glibc-2.3 (a lot work was put into thread-safeness on glibc-2.3).
--------


--------
   From
 
http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html

  "It is suggested that programs that use fork() call an exec function
  very soon afterwards in the child process, thus resetting all states. In
  the meantime, only a short list of async-signal-safe library routines
  are promised to be available."

  Note *suggested*. This is a recommendation to protect against a shoddy
  pthread-implementation. The thread specifications rule that only the
  thread calling fork() is duplicated, which initially leads to the
  recommendation (other threads holding locks aren't around to release
  them in the new execution context).

  That said, Nagios would most likely benefit greatly from a different
  means of checking things than fork()'ing twice and sending the results
  through several tiers of FIFO's. Several different methods have already
  been benchmarked. For server machines (or at least cans with a lot of
  memory and quite regularly multiple CPU's), the best way seems to be to
  create a new thread for each check to run. popen() causes a fork() and
  execve(), so that should be safe enough.

  What limits this imposes I don't know, but the NPTL library in use on
  most modern linux systems today handles 10.000 threads without barfing,
  so the limit would probably be sysconf(_SC_MAX_FILES), or ulimit -n,
  which is required by posix to be at least 256. Note that half this value
  (give or take 5 or so for stdin and such) represents the number of
  checks that can run simultaneously at any given time. When one of them
  completes another can kick in.
--------




What do you think about this ?
Should we have a specific threads nagios patch for FreeBSD ?
Nagios problem or FreeBSD problem ?
Should we switch our Nagios systems to Linux (which is very 
psychological difficult for me ...) ?

Thanks in advance for your help...  I hope we will found a solution...




More information about the freebsd-hackers mailing list