FreeBSD ports USE_XZ critical issue on low-RAM computers

[Lasse Collin]

On 2010-06-19 Matthias Andree wrote:
> We have system facilities for limiting resources, including those
> that limit virtual memory.

Limiting virtual memory with "ulimit -v" is generally not so great. It 
cripples mmap(), which can use a lot of virtual memory while using 
little actual RAM. If I made xz use mmap() for handling input files when 
possible, limiting virtual memory would have little to do with limiting 
the actual memory usage of xz: if xz mmapped a 280 MiB file that needs 
65 MiB of memory to decompress, xz would run out memory if virtual 
memory was capped to 300 MiB. Luckily for you I don't plan to use mmap() 
in xz. :-)

Perhaps FreeBSD provides a good working way to limit the amount of 
memory that a process actually can use. I don't see such a way e.g. in 
Linux, so having some method in the application to limit memory usage is 
definitely nice. It's even more useful in the compression library, 
because a virtual-memory-hog application on a busy server doesn't 
necessarily want to use tons of RAM for decompressing data from 
untrusted sources.

> For compression, it's less critical because service is degraded, not
> denied, but I'd still think -M max would be the better default. I can
> always put "export XZ_OPT=-3" in /etc/profile.d/local.sh or wherever
> it belongs on the OS of the day.

If a script has "xz -9", it overrides XZ_OPT=-3.

> I still think utilities and applications should /not/ impose
> arbitrarily lower limits by default though.

There's no multithreading in xz yet, but when there is, do you want xz 
to use as many threads as there are CPU cores _by default_? If so, do 
you mind if compressing with "xz -9" used around 3.5 GiB of memory on a 
four-core system no matter how much RAM it has?

I think it is quite obvious that you want the number of threads to be 
limited so that xz won't accidentally exceed the total amount of 
physical RAM, because then it is much slower than using fewer threads. 
Being faster is the whole point of threading anyway. Naturally doing 
unusual things is sometimes wanted so a limit can be overriden. This is 
all about the default behavior only.

In most cases, lowering the compression settings automatically is 
friendly towards the user. People easily write "xz -9" to scripts 
without thinking if they actually want that, because they are used to -9 
with gzip and bzip2. I would find it dumb to annoy users of slightly 
older hardware with _default behavior_ that puts their system to swap 
whenever such a script is ran. They can still get the swap-till-the-
morning behavior if they really want it by disabling the limit when 
compressing by using XZ_OPT.

> > Disabling the limiter completely by default doesn't seem like an
> > option, because it would only change who will be annoyed. Comments
> > are very welcome. Thanks.
> 
> It is a necessity to change it.

In short, some people find a default limit annoying and some other 
people would find lack of default limit annoying. (And most people 
probably don't care.) So the question is, which group will complain 
more; obviously I cannot make everyone happy. At this point it starts to 
look like that your group is winning. ;-) I will have to discuss with 
people in the other group before making decisions.

-- 
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode