nagios and freebsd threads issue : help please ...

[Warner Losh]

Thanks for forwarding the reply.

First, I'll note that the response contains highly charged,
inflamitory language.  I've tried my best to ignore that language and
focus on the actual issues.  Second, this posting demonstrates a
fundamental confusion between thread-safe and async-safe.  That is the
root of the problem in the communication.  Thread-safe functions are a
dime a dozen and relatively easy to write.  async-safe functions are
very rare and much harder to do useful things with.  I've tried to
explain the difference below using fgets() as an example of the
difficulties.

> fgets() must also be async-safe, since it's passed its storage-buffer
> from the calling function. It can contain races if several threads (or
> programs for that matter) tries to read FIFO's at the same time or are
> trying to store things to the same piece of memory, but that's neither
> new, strange or in any way non-obvious. Obviously, fgets() relies on
> lower-level IO code which must be thread-safe (read() in this case) on
> account of them being syscalls inside multitasking kernels.

fgets need not be async-safe, but it does need to be thread-safe.
When one fork after pthread_create, one may only call async-safe
functions.  The weaker requirements of thread safety can be shown to
not necessarily be async safe.  If two different threads call fgets(),
mutexes will keep one thread from running if the other is in the
middle of changing the FILE * internal state.  However, if that thread
is interrupted by the scheduler with the mutex held, and fork() is
called, then the new copy of the address space will still have that
mutex held.  Any attempt by this new process, with its own address
space, to acquire the lock is doomed to failure.  Since the parent and
child execute in different address spaces, there is no way for a
thread that does not exist in the child to unlock the locked mutex.


Normally this happens like so:

	Thread A				Thread B

	fgets(fp, b1, 10);
		lock fp's mutex
		copy 5 available bytes into b1
<thread scheduler interrupts here>
						fgets(fp, b2, 10)
						try lock fp's mutex
<thread scheduler puts on the pending list, maybe resuming A>
		unlock fp's mutex
	return
<thread scheduler wakes up B>
						attempt to lock finishes
						b2 can be updated
						unlock mutex.

However, in the fork case:

	Thread A				Thread B

	fgets(fp, b1, 10);
		lock fp's mutex
		copy 5 available bytes into b1
<thread scheduler interrupts here>
						fork()
	<thread A is now gone in child>
						fgets(fp, b2, 10)
						try lock fp's mutex
At this point B', the only thread in the child, will never be able to
grab this lock because A exists only in the parent and the
parent/child have independent address spaces.

While the above example is not what nagios is doing, it illustrates
the point.  There are some functions that necessarily touch global
state.  These functions need to coordinate that touching of state.  If
one of the is interrupted with locks held, then all bets are off of a
program forks and the threads holding those locks can never unlock
them.

>  >>  The list of async-signal-safe functions
>  >> are here: http://www.opengroup.org/onlinepubs/009695399/nframe.html
>  >> The restriction on fork() is here (20th bullet down):
>  >> http://www.opengroup.org/onlinepubs/009695399/nframe.html
> 
> Both of those links point to the same document, which is just the
> frameset for the navigation-frames.
> 
> For async-safe functions, this is the proper url;
> http://www.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_09.html#tag_02_09_01

This reference is for thread-safe functions.  You are confusing
thread-safe and async-safe.  The correct url for async-safe is

http://www.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_04.html#tag_02_04_03

>> The following table defines a set of functions that shall be either
>> reentrant or non-interruptible by signals and shall be
>> async-signal-safe. Therefore applications may invoke them, without
>> restriction, from signal-catching functions:
>>	<list omitted, since it has been posted before>

Notice that this list is very short, and there are many functions that
one would think should be on here, but in fact aren't.

> For the fork() specification, the doc is here;
> http://www.opengroup.org/onlinepubs/009695399/functions/fork.html
...
> "A process shall be created with a single thread. If a multi-threaded
> process calls fork(), the new process shall contain a replica of the
> calling thread and its entire address space, possibly including the
> states of mutexes and other resources. Consequently, to avoid errors,
> the child process may only execute async-signal-safe operations until
> such time as one of the exec functions is called.

Notice here it says specifically 'async-sngial-safe operations' not
'thread-safe' operations.  The standard explicitly calls attention to
the difficulties and differences between these two types of functions.

> This is funny, because nagios apparently runs properly on Linux, HPUX,
> Solaris, Irix, AIX and Tru64. To me that seems to indicate that Nagios
> is very portable indeed and that the BSD fellows somehow botched it. I
> might be wrong, but...

Just because it works doesn't make it standards conforming.

Maybe there's some simple extension that can be implemented to help
the situation.  However, the inflamitory language gets very much in
the way of having a technical discussion.

Warner