Re: libc6 Compatibility

>>>>> On Wed, 15 Jun 2022 20:26:13 +0200 (CEST), Sysadmin Lists said:
> 
> > ----------------------------------------
> > From: Martin Simmons <martin@lispworks.com>
> > Sent: Wed Jun 15 19:04:48 CEST 2022
> > To: Sysadmin Lists <sysadmin.lists@mailfence.com>
> > Cc: <freebsd-emulation@freebsd.org>, <dchagin@freebsd.org>
> > Subject: Re: libc6 Compatibility
> > 
> > 
> > >>>>> On Tue, 14 Jun 2022 21:55:58 +0200 (CEST), Sysadmin Lists said:
> > > 
> > > > ----------------------------------------
> > > > From: Dmitry Chagin <dchagin@freebsd.org>
> > > > Sent: Tue Jun 14 09:24:37 CEST 2022
> > > > To: Sysadmin Lists <sysadmin.lists@mailfence.com>
> > > > Cc: Freebsd Emulation <freebsd-emulation@freebsd.org>
> > > > Subject: Re: libc6 Compatibility
> > > > 
> > > > 
> > > > On Tue, Jun 14, 2022 at 03:26:03AM +0200, Sysadmin Lists wrote:
> > > > > Does Linux compat not work with Ubuntu's newest libc6 update?
> > > > > I used to run Brave browser from it, but now I get this error message:
> > > > > 
> > > > > $ /compat/ubuntu/opt/brave.com/brave/brave
> > > > > ELF interpreter /lib64/ld-linux-x86-64.so.2 not found, error 2
> > > > > Abort trap
> > > > > 
> > > > > The shared object is loaded in memory:
> > > > > $ ldd /opt/brave.com/brave/brave  | grep ld-
> > > > > 	/lib64/ld-linux-x86-64.so.2 (0x0000000001021000)
> > > > > 
> > > > > And it exists on the filesystem:
> > > > > $ find /compat/ubuntu/lib**/ -name ld-\* -exec ls -lh '{}' +
> > > > > -rwxr-xr-x  1 root  wheel   187K Dec 16  2020 /compat/ubuntu/lib/x86_64-linux-gnu/ld-2.31.so
> > > > > lrwxr-xr-x  1 root  wheel      10B Dec 16  2020 /compat/ubuntu/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 -> ld-2.31.so
> > > > > lrwxr-xr-x  1 root  root         34B Mar 23  2021 /compat/ubuntu/lib64/ld-linux-x86-64.so.2 -> ../lib/x86_64-linux-gnu/ld-2.31.so
> > > > > 
> > > > > This is the update that broke it:
> > > > > $ apt-get install libc6
> > > > > Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libc6 amd64 2.31-0ubuntu9.9 [2722 kB]
> > > > >
> > > > 
> > > > try ktrace it, aka ktrace -di /compat/ubuntu/opt/brave.com/brave/brave
> > > > then kdump -HAR -m 128 > brave.log
> > > > 
> > > > 
> > > 
> > > Thanks, Dmitry. That helped me find the problem. Here's the full log:
> > > 
> > > $ cat brave.log 
> > >   8184 101222 ktrace   0.000000 RET     F64  ktrace 0
> > >   8184 101222 ktrace   0.000014 CALL   F64  execve(0x7fffffffeda3,0x7fffffffeac0,0x7fffffffead0)
> > >   8184 101222 ktrace   0.000003 NAMI  F64  "/compat/ubuntu/opt/brave.com/brave/brave"
> > >   8184 101222 ktrace   0.055726 NAMI  F64  "/compat/ubuntu/lib64/ld-linux-x86-64.so.2"
> > >   8184 101222 ktrace   0.000032 NAMI  F64  "/lib64/ld-linux-x86-64.so.2"
> > > 
> > > And here's an updated `ls' printout after upgrading libc6:
> > > $ find /compat/ubuntu/lib**/ -name ld-\* -exec ls -lh '{}' +
> > > -rwxr-xr-x  1 root  wheel   187K Apr  6 18:24 /compat/ubuntu/lib/x86_64-linux-gnu/ld-2.31.so
> > > lrwxr-xr-x  1 root  wheel      10B Apr  6 18:24 /compat/ubuntu/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 -> ld-2.31.so
> > > lrwxr-xr-x  1 root  wheel      32B Apr  6 18:24 /compat/ubuntu/lib64/ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.31.so
> > > 
> > > The new .so is missing 'previous directory' dots in the symlink path.
> > 
> > It looks like the link with the dots was made by hand in 2021, presumably to
> > fix this.
> 
> That might be the case. I remember having to make changes to the shared object 
> before.  The question then becomes, why only this update to libc6 and not 
> previous updates?

Based on the file dates, I suspect there were no previous updates to libc6
installed since Dec 16 2020.


> > > I thought Linux compat was designed to search compat.linux.emul_path
> > > for shared libs first, but apparently it works differently than that.
> > 
> > What does linux compat do in general with absolute symlinks?  Does it check
> > the target with compat.linux.emul_path prepended?
> 
> That's what the documentation says:
> "Linux mode dynamically reroots lookups. This is, in effect, equivalent to
> union to file system mounts. First, an attempt is made to look up the file in
> /compat/linux/original-path. If that fails, the lookup is done in
> /original-path. This makes sure that binaries that require other binaries can
> run."
> https://docs.freebsd.org/en/books/handbook/linuxemu/#linuxemu-advanced
> 
> Looking at the installer script, it's clear this is a known issue:
> $ sed -n '/fix_ld_path(/,/^$/p' /usr/local/share/linux-browser-installer/linux-browser-installer
> fix_ld_path()
> {
>         (cd ${chroot_path}/lib64 && \
>                 (unlink ./ld-linux-x86-64.so.2; \
>                         ln -s ../lib/x86_64-linux-gnu/ld-${ld_version}.so \
>                         ld-linux-x86-64.so.2))
> }
> 
> So, either:
> 1. The documentation is wrong.
> 2. It's a bug.

Looking at how it is implemented, the rerooting applies to the argument to the
Linux system call before it calls the code that implements the FreeBSD system
call to do the actual work.  I think the problem here is that symlinks are
resolved by FreeBSD system call code (the NAMI operation), but that doesn't do
any rerooting.

I don't know if that is a bug or a necessary limitation.

__Martin