VirtualBox (sometimes) not starting/hanging after recent openssl updates

[Christoph Moench-Tegeder]

Hi,

for reference, prior reports (not mine):
https://lists.freebsd.org/pipermail/freebsd-emulation/2015-March/012381.html
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198952

I got bitten by the very same problem - "NS_ERROR_FACTORY_NOT_REGISTERED"
when running VirtualBox "as usual", "VERR_INVALID_POINTER" and "environment
corrupt" when using a DEBUG-enabled VirtualBox.

As I was pretty sure I had a working VirtualBox 4.3.26 before, I tracked
all the changes since the last successful VirtualBox use, and the only
"significant" thing was that I upgraded my base system for
FreeBSD-SA-15:06.openssl (and the fix for that, so I'm up-to-date with
releng/10.1 (r280275).

After some time of poking the source and adding more debug statements
and asserts (debugging suid code...), I got the following backtrace in
my core dump (abbreviated):
(gdb) bt
#0  0x00000008033257c5 in OPENSSL_ia32_cpuid ()
   from /usr/local/lib/libcrypto.so.8
#1  0x0000000805f88eb9 in ?? () from /lib/libcrypto.so.7
#2  0x0000000805e8f84e in _init () from /lib/libcrypto.so.7
#3  0x00007fffffffc760 in ?? ()
#4  0x000000080060e6bf in ?? () from /libexec/ld-elf.so.1
#5  0x0000000800612d87 in ?? () from /libexec/ld-elf.so.1
#6  0x000000080060fad3 in ?? () from /libexec/ld-elf.so.1
#7  0x00000000004042d9 in supR3HardenedMainInitRuntime (fFlags=3)

Having different versions of a library calling each other (base openssl
vs. ports openssl) is most certainly a bad thing.
More digging showed that the libcryptos where pulled in by libcurl.so.4.
That (curl) in turn links against ports openssl when found (there's
even code in the Makefile to stop the build when one tries forcing the
use of base openssl), but happily pulls in base openssl via it's
dependencies. One instance of the base openssl was found in librtmp
and could be solved by recompiling that. The other problem was with
the use of "GSSAPI_BASE" (base system GSSAPI) in curl - which is the
default, according to a curl-less machine in the corner.

In the short run, people affected by the VirtualBox problem should
check if /usr/local/lib/virtualbox/VBoxRT.so links against both
(base and ports) libcryptos (use ldd) - if so, check libcurl (and if
that does not help, the other shared libraries) for libcrypto mixup.

In the slightly longer run - how do we want to fix this? I can imagine
preventing the use of GSSAPI_BASE when ports openssl is detected - there's
similar code already for preventing WITH_OPENSSL_BASE usage when
ports openssl is installed. OTOH this will not help in case the "wrong"
openssl is pulled in via some other dependency (librtmp in my case).

Regards,
Christoph

-- 
Spare Space