Stuck CLOSED sockets / sshd / zombies...

[Chris]

> On Fri, Apr 11, 2014 at 03:57:00PM +0100, Karl Pielorz wrote:
>>
>>
>> --On 11 April 2014 17:15 +0300 Konstantin Belousov <kostikbel at gmail.com>
>> wrote:
>>
>> > So my suspicious idea seems to be true. From the ldd output, libc
>> > appears before libthr in the global order, so libc sigaction() symbol
>> > is resolved before libthr interposer. The result is that libthr wrapper
>> > thr_sighandler() for the signal handlers is not installed as the
>> > recepient of the kernel signal, which prevents libthr locks for rtld
>> > from working properly.
>>
>> Ok, I can just about follow that ;)
>>
>> > To confirm or deny my theory, please apply the patch below, in addition to
>> > the previous patch, and rebuild sshd only,
>> ># cd src/secure/usr.sbin/sshd && make clean all install
>> > The patch tilts the order of initialization, for my build I got
>> > sandy% ldd /usr/sbin/sshd
>> > ...
>> >         libz.so.6 => /lib/libz.so.6 (0x802f0d000)
>> >         libthr.so.3 => /lib/libthr.so.3 (0x803123000)
>> >         libc.so.7 => /lib/libc.so.7 (0x803348000)
>> >         libldns.so.5 => /usr/lib/private/libldns.so.5 (0x8036d1000)
>> >         libmd.so.6 => /lib/libmd.so.6 (0x803926000)
>> > which could be enough to prevent the bug.
>> >
>> > Please retest and report.
>>
>> Ok, patched the makefile - rebuilt / installed sshd restarted (which has
>> the same initialisation order as yours above), it and ran the security scan
>> against it.
>>
>> *This does indeed seem to fix the problem*
>>
>> The scan completes, and there are no stuck 'urdlck' sshd's - and no socket
>> sitting around in CLOSE_WAIT or CLOSED - thanks! :)
>>
>> I re-ran the scan a couple of times more to be sure, with the same result -
>> no zombies or anything.
> Great.
>
>>
>> Is this situation likely to be repeated anywhere else on the system? Or is
>> it likely just to be specific to sshd?
> Well.
>
> The issue with libthr so relying on interposition of libc has already bitten
> us more than once.  The biggest practical consequence of it is that libthr
> cannot be dynamically loaded, it must be linked to the main binary for the
> whole construct to work.  This means that any program big enough to load
> plugins at runtime must link to libthr if it might need to load plugin
> linked to libthr.  Recently, perl and other programs from ports started
> doing just that.
>
> But this is first time I see interposing so broken due to wrong linking
> order, esp. in the base system.
>
> The correct solution is to merge libthr into libc. Some neccessary
> preparations were already done, but the main work did not started yet.
> This is huge efforts, and it probably should be coordinated with some
> other ABI changes planed for libthr to support process-shared locks.
>

This may be wishful thinking, or I might have even overlooked the obvious.
Would it make any sense to create 2 versions of libthr, one with libc,
and one without? At least that would overcome the /possible/ ordering
issue associated with this sort of thing. Then it's simply a matter of
choosing which one to load.

--Chris

> Anyway, for now, this patch, possibly enhanced to only link with
> libpthread when kerberos is used, should be good enough.
>