zfs + uma

[Andre Oppermann]

On 17.09.2010 10:14, Andriy Gapon wrote:
>
> I've been investigating interaction between zfs and uma for a while.
> You might remember that there is a noticeable fragmentation in zfs uma zones
> when uma use is not enabled for actual data/metadata buffers.
>
> I also noticed that when uma use is enabled for data/metadata buffers
> (zio.use_uma=1) amount of memory reserved in free items of zfs uma zones becomes
> really huge.  And this is despite the fact that the vast majority of the
> data/metadata zone have items with sizes that are multiples of page size.
> This couldn't really be because of fragmentation.
>
> Further checks show that the free items are accumulated in per-cpu cache
> buckets.  uz_count for those buckets starts with 1, but over time, during bursts
> of activity, it grows up to maximum of 128.
> Problem with those buckets is that they are not drained on low memory conditions
> and uz_count never goes down.
>
> So, after a while, I observe about 300 free items (on a mere two core system)
> cached in 4 per-cpu buckets for a single zone with 128KB item size.
> That's 30MB right there.
> For all data and metadata zones the number goes as high as 500MB on my machine
> with 4GB physical RAM.
> This seems like a bit too much to me.
>
> Although keeping free items around improves performance, it does consume memory
> too.  And the fact that that memory is not freed on lowmem condition makes the
> situation worse.

Interesting.  We may run into related issues with excessive mbuf
(cluster) caching in the per-cpu buckets as well.

Having a general solutions for that is appreciated.  Maybe the size
of the free per-cpu buckets should be specified when setting up the
UMA zone.  Of certain frequently re-used elements we may want to
cache more, other less.

-- 
Andre