Why does adding /usr/lib32 to LD_LIBRARY_PATH break 64-bit binaries?

[Nate Eldredge]

On Thu, 23 Oct 2008, Alexander Sack wrote:

> Alright, well I found some weirdness:
>
> [root at hagen ~]# export LD_LIBRARY_PATH=/usr/bin:/usr/lib:/usr/lib32:/usr/lib64
> [root at hagen ~]# LD_DEBUG=1 ls
> /libexec/ld-elf.so.1 is initialized, base address = 0x800506000
> RTLD dynamic = 0x80062ad78
> RTLD pltgot  = 0x0
> processing main program's program header
> Filling in DT_DEBUG entry
> lm_init("(null)")
> loading LD_PRELOAD libraries
> loading needed objects
> Searching for "libutil.so.5"
>  Trying "/usr/bin/libutil.so.5"
>  Trying "/usr/lib/libutil.so.5"
>  Trying "/usr/lib32/libutil.so.5"
> loading "/usr/lib32/libutil.so.5"
> /libexec/ld-elf.so.1: /usr/lib32/libutil.so.5: unsupported file layout
>
> That's because libutil.so.5 does not exist in /usr/lib only in /lib.
> The /usr/lib directory has:
>
> [root at hagen ~]# ls -l  /usr/lib/libutil*
> -r--r--r--  1 root  wheel  100518 Aug 21  2007 /usr/lib/libutil.a
> lrwxrwxrwx  1 root  wheel      17 Sep 11 11:44 /usr/lib/libutil.so ->
> /lib/libutil.so.5
> -r--r--r--  1 root  wheel  103846 Aug 21  2007 /usr/lib/libutil_p.a
>
> So rtld is looking for major number 5 of libutil, without the standard
> /lib in my LD_LIBRARY_PATH it searches /usr/lib, doesn't find it but:
>
> [root at hagen ~]# ls -l  /usr/lib32/libutil*
> -r--r--r--  1 root  wheel  65274 Aug 21  2007 /usr/lib32/libutil.a
> lrwxrwxrwx  1 root  wheel     12 Sep 11 11:45 /usr/lib32/libutil.so ->
> libutil.so.5
> -r--r--r--  1 root  wheel  46872 Aug 21  2007 /usr/lib32/libutil.so.5
> -r--r--r--  1 root  wheel  66918 Aug 21  2007 /usr/lib32/libutil_p.a
>
> And whalah, I'm broke since there is a libutil.so.5 in there.
>
> So my question to anyone out there, WHY does /usr/lib32 contain major
> numbers but /usr/lib does not?  This seems like a bug to me (FreeBSD
> 7.0-RELEASE is the same) or at least a dubious design decision.

I think the distinction is this.  rtld is looking for libutil.so.5 (with 
version number).  This file has to be in /lib, in the root filesystem, so 
that programs can run before /usr is mounted.

libutil.so on the other hand is not searched for by rtld, but by ld 
(driven by cc), when the program is built.  /usr/lib is the traditional 
place for it to search; I'm not sure if it searches /lib at all.  In the 
case of static libraries, /usr/lib is certainly the right place for 
libutil.a to go, so having libutil.so there makes sense in my mind.

I think your best bet is to dig into whatever is setting LD_LIBRARY_PATH 
and get it set correctly.  Remove /usr/lib32 or at least ensure that /lib 
is searched first.  Trying to change rtld's behavior is not the right 
approach, IMHO.

-- 

Nate Eldredge
neldredge at math.ucsd.edu