svn commit: r270759 - in head/sys: cddl/compat/opensolaris/kern cddl/compat/opensolaris/sys cddl/contrib/opensolaris/uts/common/fs/zfs vm

[John Baldwin]

On Wednesday, September 10, 2014 08:30:58 PM Slawa Olhovchenkov wrote:
> On Wed, Sep 10, 2014 at 10:44:46AM -0400, John Baldwin wrote:
> > On Tuesday, September 09, 2014 02:25:18 PM Slawa Olhovchenkov wrote:
> > > On Mon, Sep 08, 2014 at 11:17:51AM -0400, John Baldwin wrote:
> > > > On Sunday, September 07, 2014 04:56:49 PM Slawa Olhovchenkov wrote:
> > > > > PS: very bad that 'data limit' don't anymore reflect application
> > > > > memory consumer. and very small application can adapt to 'no memory'
> > > > > from system.
> > > > 
> > > > You can use RLIMIT_AS instead of RLIMIT_DATA.  jemalloc can also be
> > > > configured to use sbrk(), though I think there's no way to prevent it
> > > > from falling back to mmap(MAP_ANON) if sbrk() fails.
> > > 
> > > Formally you are right. Realy this is scorn. And I don't know how
> > > rightly. May be account in RLIMIT_DATA MAP_ANON? Anyway firefox don't
> > > run GC if malloc fail and this is bad.
> > 
> > It was not intended as scorn, simply stating what options you have
> 
> Hmm, this is may bad english and try to translate some russian sentence.
> I am try to do some tuning for satisfaction me.
> You proposal do some limiting, but don't delight.

Ok.
 
> > available as alternatives to what you are asking for.  The Firefox
> > behavior certainly seems unfortunate. :(  However, using RLIMIT_AS
> > for that should work as the mmap() call malloc() makes will fail
> > causing malloc() to fail.  However, you need RLIMIT_AS to be a bit
> > larger than what you would set RLIMIT_DATA to.
> > 
> > As to having MAP_ANON honor RLIMIT_DATA, I'm not sure how well that
> > would work.  I assume that was discussed as an option in the previous
> > threads on the topic of RLIMIT_DATA being useless with jemalloc.
> > I don't recall it though.  Perhaps Alan can speak to whether or not
> > that is a good idea?
> 
> What is sense of RLIMIT_DATA (as I understand)?
> This is signaling to application of memory consumption.

Yes, traditionally it only affected malloc() and not other portions of the 
address space like the stack (RLIMIT_STACK) or anything in shared libraries 
(including text/data/bss).

> Limiting RSS don't visible to application.

Yes, RLIMIT_RSS is not often useful because it is not synchronous with the
RSS changing.

> Application must change behaviour when reach limit (run GC, switch
> algorithm and etc.).
> But mmap of big data file for scaning and processing don't touch this
> limit.
> Must be mmap of some temoprary file touch this limit? I don't know.
> Must be MAP_ANON touch this limit? I think yes.
> How to make distinction of mmap data file for processing and mmap
> temporary file for bypass this limit? I don't know.

Consider also mmap() of shm_open(SHM_ANON).

> And again, most application don't handle correctly NULL of malloc().
> I think this is depreciate of all.

This is true, but it seems Firefox might, and you could set RLIMIT_AS
for Firefox in particular so it gets adequate feedback.  I do think if
you can determine the "correct" value for RLIMIT_AS for Firefox that
mmap(MAP_ANON) would fail and result in NULL from malloc().

> PS: question about jemalloc. How I can stop jemalloc to give back
> memory to OS for some application?

You would have to hack pages_purge() in chunk_mmap.c to always return true and 
not call madvise().

-- 
John Baldwin