libarchive, lzma, and xz interaction

[Kevin Oberman]

On Fri, Jul 1, 2011 at 11:32 PM, b. f. <bf1783 at googlemail.com> wrote:
>> On Fri, Jul 1, 2011 at 7:19 PM, Kevin Oberman <kob6558 at gmail.com> wrote:
>> > I'm trying to understand the problems I am having on some systems
>> > regarding libarchive, lzma, and xz.
>> > I have an 8-Stable system updated yesterday. As far as I can tell,
>> > libarchive does include the lzma stuff
>> > from libzma. At least I see the references. But several ports seem to
>> > still pull in xz-5.0.1 and link to it.
>> > This has a wonderful potential to cause library symbol conflicts. I get:
>> > /usr/lib/libarchive.so: undefined reference to `lzma_stream_encoder at XZ_5.0'
>> > /usr/lib/libarchive.so: undefined reference to `lzma_alone_decoder at XZ_5.0'
>> > /usr/lib/libarchive.so: undefined reference to `lzma_memusage at XZ_5.0'
>> > /usr/lib/libarchive.so: undefined reference to `lzma_stream_decoder at XZ_5.0'
>> > /usr/lib/libarchive.so: undefined reference to `lzma_code at XZ_5.0'
>> > /usr/lib/libarchive.so: undefined reference to `lzma_end at XZ_5.0'
>> > /usr/lib/libarchive.so: undefined reference to `lzma_lzma_preset at XZ_5.0'
>> > /usr/lib/libarchive.so: undefined reference to `lzma_alone_encoder at XZ_5.0'
>> >
>> > ldd shows libarchive linked against liblzma.so.5 and an objdump of the
>> > dynamic symbols from liblzma.so.5
>> > shows the "undefined symbols" defined with the XZ_5.0 version, so I am
>> > mystified. It looks o me like it is
>> > there. Is confusion with xz-5.0.1 causing this? Should get rid of it?
>> > Even so, I don't understand why the
>> > loader is claiming that these symbols are undefined when they seem to
>> > be defined as far as I can tell.
>> > 0000000000007c60 g    DF .text  0000000000000084  XZ_5.0
>> > lzma_stream_encoder
>> >
>> > Any clues to what i happening would be greatly appreciated!
>
> Nothing mysterious here:
>
> src/lib/libarchive/archive_write_set_compression_xz.c uses some code
> from the base system /usr/lib/liblzma.so shared library, from
> src/lib/liblzma, which is derived from the base system xz 5.0.1,  in
> src/contrib/xz.  Hence the symbols are present, with the symbol
> versions that you noted, but undefined in libarchive, because it is
> obtaining them from liblzma at run-time.  It nudges rtld(1) to do this
> in the usual way, via it's DT_NEEDED tag for that library:
>
> objdump -x /usr/lib/libarchive.so | egrep 'NEEDED.*lzma'
>
>> Replying to myself, I re-built all programs that depended on xz and
>> then deleted the port. Now everything works fine. I still don't
>> completely understand what I was seeing, but it is working, now.
>
> Perhaps you recently updated your base system, but were using old
> ports or packages?   With up-to-date ports and a recent version of
> 8-stable you should not have been able to build the archivers/xz port,
> even if other ports mistakenly tried to drag it in, because of the
> following lines in the ports/archivers/xz Makefile:
>
>
> .if ${OSVERSION} >= 900012 || (${OSVERSION} < 900000 && ${OSVERSION} >= 800505)
> IGNORE= is already in the base system
> .endif
>
> Of course, you could have overridden this, by setting NO_IGNORE.

Thanks! Now I understand what was happening much better. It does make sense.

And you were right about the sequence o events. Due to a bug in the
atkbd driver in 8.1 and 8.2 I installed 8.0-RELEASE and, after
installing a few critical ports (which must have pulled in xz), I
csuped RELENG_8, I applied the required patch (which has now been
MFCed), and built a stable system. Unfortunately, I now had xz and got
into the problems I reported.

I am a bit disturbed to note that there is no entry in either
/usr/src/UPDATING or /usr/ports/UPDATING about lzma moving into the
base system and the need to re-install ports that had depended on the
xz port and delete xz. While googling for the problem I found many
reports of the problem with no solutions. A note in UPDATING (in this
case, probably both of them, but at least /usr/src) could have made
them easily resolved.

Thanks for taking the time to educate me. I've been working with
FreeBSD for several years, but I still don't know a great many things
about it.
-- 
R. Kevin Oberman, Network Engineer - Retired
E-mail: kob6558 at gmail.com