FreeBSD ports USE_XZ critical issue on low-RAM computers

Matthias Andree mandree at
Sun Jun 20 13:47:57 UTC 2010

Am 19.06.2010 15:41, schrieb Lasse Collin:

> Perhaps FreeBSD provides a good working way to limit the amount of 
> memory that a process actually can use. I don't see such a way e.g. in 
> Linux, so having some method in the application to limit memory usage is 
> definitely nice. It's even more useful in the compression library, 
> because a virtual-memory-hog application on a busy server doesn't 
> necessarily want to use tons of RAM for decompressing data from 
> untrusted sources.

Even there the default should be "max", and the library SHOULD NOT
second-guess what trust level of data the application might to process
with libxz's help.

Expose the limiter interface in the API if you want, but particularly
for the library in particular, any other default than "unlimited memory"
is a nuisance.  And there's still an application, and unlike the xz
library, the application should know what kind of data from what sources
it is processing, and if - for instance - a virus inspector wants to
impose memory limits and quarantine an attachment with what looks like
an zip bomb.

Typically, after the advent of KDE, GNOME, XFCE, and thereabouts with
all their graphical tools, and people hardly use command line tools
unless they know exactly what they're doing - and it's not as though
xz's behaviour were prone to causing permanent damage somewhere, so it's
OK if less skilled users find out the hard way that they need to read

I was surprised because xz has somewhat left the traditional UNIX way,
which was try exactly as little and as hard as you were told, until you
bump into a brick wall (permission denied on some file, or memory
allocation failed, or similar).
Don't try to be nice unless you're asked to.
Don't try to ask questions unless you're asked to be "interactive".

All your defaults limit make using the utility unnecessarily hard, make
it harder to explain, because the default limits are surprising and
cause self-made failures.

And I think many people would just lay xz or the library aside when
figuring they need to do this and that and foo and bar and torture a
black cat, swinging it over my head, and dance strange figures in the
sewers in a full moon night, just so that xz or the library finally
condescend to decompressing a file. I am exaggerating here, but please,
don't make me jump through hoops with my application or script.

I'd say a typical application wants to call xzopen() and decompress a
file, and if it wants to impose limits it will use setrlimit() or
perhaps xz_set_memory_limit in addition beforehand.

I do fear that this will actually hamper, not foster, adoption of the xz

>> For compression, it's less critical because service is degraded, not
>> denied, but I'd still think -M max would be the better default. I can
>> always put "export XZ_OPT=-3" in /etc/profile.d/ or wherever
>> it belongs on the OS of the day.
> If a script has "xz -9", it overrides XZ_OPT=-3.

I know. This isn't a surprise for me. The memory limiting however is.
And the memory limiting overrides xz -9 to something lesser, which may
not be what I want either.

>> I still think utilities and applications should /not/ impose
>> arbitrarily lower limits by default though.
> There's no multithreading in xz yet, but when there is, do you want xz 
> to use as many threads as there are CPU cores _by default_? If so, do 
> you mind if compressing with "xz -9" used around 3.5 GiB of memory on a 
> four-core system no matter how much RAM it has?

Multithreading in xz is worth discussion if the tasks can be
parallelized, which is apparently not the case.  You would be
duplicating effort, because we have tools to run several xz on distinct
files at the same time, for instance BSD portable make or GNU make with
a "-j" option.

> I think it is quite obvious that you want the number of threads to be 
> limited so that xz won't accidentally exceed the total amount of 
> physical RAM, because then it is much slower than using fewer threads. 

This tells me xz cannot fully parallelize its effort on the CPUs, and
should be single-threaded so as not to waste the parallelization overhead.

> Being faster is the whole point of threading anyway. Naturally doing 
> unusual things is sometimes wanted so a limit can be overriden. This is 
> all about the default behavior only.

Yes, and I consider the default behaviour to be "getting in my way" and
disturbing.  No other compression tool I know would ever spend that much
thought on its working environment. All others will only fail if it's
"physically" impossible to complete the job, and otherwise just grind away.

> In most cases, lowering the compression settings automatically is 
> friendly towards the user. People easily write "xz -9" to scripts 
> without thinking if they actually want that, because they are used to -9 
> with gzip and bzip2.

-9 is quite slow in bzip2, in gzip computers have become fast enough so
that -9 hardly hurts today, but I recall times where I thought twice
before using gzip -9 rather than gzip -3.  People know there is a price
tag attached, else the whole option system would be useless and --best
were the only option.

> I would find it dumb to annoy users of slightly
> older hardware with _default behavior_ that puts their system to swap 
> whenever such a script is ran. They can still get the swap-till-the-
> morning behavior if they really want it by disabling the limit when 
> compressing by using XZ_OPT.

This is really xz developing a life of its own.


If I specify -9 or --best, but no memory option, that means "compress as
hard as you can".

Instead, xz assumes an implicit default memory limit, so the -9 gets
degraded to -5, -2, -1, ... in an somewhat surprising manner, because
depending on which computer I run it on, -9 might mean -9, or -6 on
another computer, or -1 on a third.

That is what I'd call a nasty surprise - xz overrides my command line
option.  I would propose that with -9 and without -M option, that it
tries to allocate memory, and if it fails, it can still suggest to use
the -M option or a lower -[0-8] option so I know how to proceed.

> In short, some people find a default limit annoying and some other 
> people would find lack of default limit annoying. (And most people 
> probably don't care.) So the question is, which group will complain 
> more; obviously I cannot make everyone happy. At this point it starts to 
> look like that your group is winning. ;-) I will have to discuss with 
> people in the other group before making decisions.

The real thing is that the xz software does things that go beyond the
options. This has been a negative surprise to me.

I also like to recall my earlier argument that xz is a low-level tool,
and it's mixing high-level features in. This is astonishing.

I think that some of the defaults you've set were trying to address
usability concerns, and there are other ways to achieve this usability.
Often it's an alternative proposal together with a diagnostic that
suffices. Helping towards self-aid, in a way.

I hope - egoistically - that xz will lean towards easier use in
infrastructure (think build systems or applications using libraries),
rather than assisting newbies. The more consistent the world around
Unix-compatible and -environment tools is, the easier people will learn.
It keeps the documentation simpler because there's not so many ifs and
buts, and in the end I think it will pay off to change the default.

Best regards

More information about the freebsd-ports mailing list