libcrypto.so.111 linked binaries SIGSEGV (in bhyve guest)

[Harry Schmalzbauer]

Hello,

I'm tryint to upgrade a bhyve guest from stable/11 to stable/12.

pkg(8) for example crashes with signal 11.

I looked for other binaries affected by
ldd /usr/sbin/* | & grep 'signal 11$'
wich gives
/usr/sbin/auditdistd: signal 11
/usr/sbin/bhyve: signal 11
/usr/sbin/bsnmpd: signal
/usr/sbin/gssd: signal 11
/usr/sbin/hostapd: signal 11
/usr/sbin/iprop-log: signal 11
/usr/sbin/keyserv: signal 11
/usr/sbin/kstash: signal 11
/usr/sbin/ktutil: signal 11
/usr/sbin/local-unbound: signal 11
/usr/sbin/local-unbound-anchor: signal 11
/usr/sbin/local-unbound-checkconf: signal 11
/usr/sbin/local-unbound-control: signal 11
/usr/sbin/ntp-keygen: signal 11
/usr/sbin/ntpd: signal 11
/usr/sbin/ntpdate: signal 11
/usr/sbin/ntpdc: signal 11
/usr/sbin/pkg: signal 11
/usr/sbin/ppp: signal 11
/usr/sbin/sntp: signal 11
/usr/sbin/sshd: signal 11
/usr/sbin/tcpdump: signal 11
/usr/sbin/uefisign: signal 11
/usr/sbin/wpa_supplicant: signal 11

They all seem to have in common beeing linked against 
'/lib/libcrypto.so.111'

truss /usr/sbin/auditdistd
:
close(3)                                         = 0 (0x0)
openat(AT_FDCWD,"/lib/libcrypto.so.111",O_RDONLY|O_CLOEXEC|O_VERIFY,00) 
= 3 (0x3)
fstat(3,{ mode=-r--r--r-- ,inode=150033332,size=3006464,blksize=4096 }) 
= 0 (0x0)
mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 
34362249216 (0x800265000)
mmap(0x0,3104768,PROT_NONE,MAP_GUARD,-1,0x0)     = 34362347520 (0x80027d000)
mmap(0x80027d000,1138688,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0) 
= 34362347520 (0x80027d000)
mmap(0x800393000,1757184,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x116000) 
= 34363486208 (0x800393000)
mmap(0x800540000,196608,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x2c3000) 
= 34365243392 (0x800540000) SIGNAL 11 (SIGSEGV) code=SEGV_ACCERR 
trapno=12 addr=0x80056f790
process killed, signal = 11 (core dumped)

I have no idea how to analyze further or what the reason could be (like 
mentioned, all binaries listed dump core after opening lib/libcrypto.so.111

gdb shows:
Core was generated by `/usr/sbin/auditdistd'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libutil.so.9...Reading symbols from 
/usr/lib/debug//lib/libutil.so.9.debug...done.
done.
Loaded symbols for /lib/libutil.so.9
Reading symbols from /libexec/ld-elf.so.1...Reading symbols from 
/usr/lib/debug//libexec/ld-elf.so.1.debug...done.
done.
Loaded symbols for /libexec/ld-elf.so.1
#0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
     at 
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
5624                    ((char *)dest)[i] = c;
(gdb) bt
#0  memset (dest=0x80056f790, c=0, len=<value optimized out>)
     at 
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:5624
#1  0x0000000800235b07 in map_object (fd=3, path=0x800246140 
"/lib/libcrypto.so.111",
     sb=0x7fffffffd4a8)
     at 
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/map_object.c:249
#2  0x0000000800230806 in load_object (name=0x201dba "libcrypto.so.111", 
fd_u=-1,
     refobj=0x800248000, flags=<value optimized out>)
     at 
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2493
#3  0x0000000800229972 in _rtld (sp=<value optimized out>, 
exit_proc=0x7fffffffea30,
     objp=0x7fffffffea38)
     at 
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/rtld.c:2315
#4  0x0000000800228019 in .rtld_start ()
     at 
/usr/local/share/deploy-tools/RELENG_12/src/libexec/rtld-elf/amd64/rtld_start.S:39
#5  0x0000000000000000 in ?? ()
Current language:  auto; currently minimal

Any help highly appreciated.

This is with a live CD (amd64), compiled with stable/12 from today (so 
clang 7.01).
The bhyve guest has 2GB hardwired and ran stable/11 beforehand, which 
compiled the live CD.
bhyve host is 11.2.  But that shouldn't play a role, does it?

-harry