FreeBSD ports USE_XZ critical issue on low-RAM computers
lasse.collin at tukaani.org
Sat Jun 19 13:41:25 UTC 2010
On 2010-06-19 Matthias Andree wrote:
> We have system facilities for limiting resources, including those
> that limit virtual memory.
Limiting virtual memory with "ulimit -v" is generally not so great. It
cripples mmap(), which can use a lot of virtual memory while using
little actual RAM. If I made xz use mmap() for handling input files when
possible, limiting virtual memory would have little to do with limiting
the actual memory usage of xz: if xz mmapped a 280 MiB file that needs
65 MiB of memory to decompress, xz would run out memory if virtual
memory was capped to 300 MiB. Luckily for you I don't plan to use mmap()
in xz. :-)
Perhaps FreeBSD provides a good working way to limit the amount of
memory that a process actually can use. I don't see such a way e.g. in
Linux, so having some method in the application to limit memory usage is
definitely nice. It's even more useful in the compression library,
because a virtual-memory-hog application on a busy server doesn't
necessarily want to use tons of RAM for decompressing data from
> For compression, it's less critical because service is degraded, not
> denied, but I'd still think -M max would be the better default. I can
> always put "export XZ_OPT=-3" in /etc/profile.d/local.sh or wherever
> it belongs on the OS of the day.
If a script has "xz -9", it overrides XZ_OPT=-3.
> I still think utilities and applications should /not/ impose
> arbitrarily lower limits by default though.
There's no multithreading in xz yet, but when there is, do you want xz
to use as many threads as there are CPU cores _by default_? If so, do
you mind if compressing with "xz -9" used around 3.5 GiB of memory on a
four-core system no matter how much RAM it has?
I think it is quite obvious that you want the number of threads to be
limited so that xz won't accidentally exceed the total amount of
physical RAM, because then it is much slower than using fewer threads.
Being faster is the whole point of threading anyway. Naturally doing
unusual things is sometimes wanted so a limit can be overriden. This is
all about the default behavior only.
In most cases, lowering the compression settings automatically is
friendly towards the user. People easily write "xz -9" to scripts
without thinking if they actually want that, because they are used to -9
with gzip and bzip2. I would find it dumb to annoy users of slightly
older hardware with _default behavior_ that puts their system to swap
whenever such a script is ran. They can still get the swap-till-the-
morning behavior if they really want it by disabling the limit when
compressing by using XZ_OPT.
> > Disabling the limiter completely by default doesn't seem like an
> > option, because it would only change who will be annoyed. Comments
> > are very welcome. Thanks.
> It is a necessity to change it.
In short, some people find a default limit annoying and some other
people would find lack of default limit annoying. (And most people
probably don't care.) So the question is, which group will complain
more; obviously I cannot make everyone happy. At this point it starts to
look like that your group is winning. ;-) I will have to discuss with
people in the other group before making decisions.
Lasse Collin | IRC: Larhzu @ IRCnet & Freenode
More information about the freebsd-ports