ZFS and mem management

Wed Feb 15 10:21:32 UTC 2012

2012/2/15 Pavlo <devgs at ukr.net>:
>
>
>
> Hello.
>
> We have an issue with memory management on FreeBSD and i suspect it is
> related to FS.
> We are using ZFS, here quick stats:
>
>
> zpool status
> pool: disk1
> state: ONLINE
> scan: resilvered 657G in 8h30m with 0 errors on Tue Feb 14 21:17:37 2012
> config:
>
> NAME            STATE     READ WRITE CKSUM
> disk1           ONLINE       0     0     0
> mirror-0      ONLINE       0     0     0
> gpt/disk0   ONLINE       0     0     0
> gpt/disk1   ONLINE       0     0     0
> gpt/disk2     ONLINE       0     0     0
> gpt/disk4     ONLINE       0     0     0
> gpt/disk6     ONLINE       0     0     0
> gpt/disk8     ONLINE       0     0     0
> gpt/disk10    ONLINE       0     0     0
> gpt/disk12    ONLINE       0     0     0
> mirror-7      ONLINE       0     0     0
> gpt/disk14  ONLINE       0     0     0
> gpt/disk15  ONLINE       0     0     0
>
> errors: No known data errors
>
> pool: zroot
> state: ONLINE
> scan: resilvered 34.9G in 0h11m with 0 errors on Tue Feb 14 12:57:52 2012
> config:
>
> NAME          STATE     READ WRITE CKSUM
> zroot         ONLINE       0     0     0
> mirror-0    ONLINE       0     0     0
> gpt/sys0  ONLINE       0     0     0
> gpt/sys1  ONLINE       0     0     0
>
> errors: No known data errors
>
> ------------------------------------------------------------------------
>
> System Memory:
>
> 0.95%    75.61    MiB Active,    0.24%    19.02    MiB Inact
> 18.25%    1.41    GiB Wired,    0.01%    480.00    KiB Cache
> 80.54%    6.24    GiB Free,    0.01%    604.00    KiB Gap
>
> Real Installed:    8.00    GiB
> Real Available:    99.84%    7.99    GiB
> Real Managed:    96.96%    7.74    GiB
>
> Logical Total:    8.00    GiB
> Logical Used:    21.79%    1.74    GiB
> Logical Free:    78.21%    6.26    GiB
>
> Kernel Memory:    1.18    GiB
> Data:    99.05%    1.17    GiB
> Text:    0.95%    11.50    MiB
>
> Kernel Memory Map:    4.39    GiB
> Size:    23.32%    1.02    GiB
> Free:    76.68%    3.37    GiB
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------
> ZFS Subsystem Report    Wed Feb 15 10:53:03 2012
> ------------------------------------------------------------------------
>
> System Information:
>
> Kernel Version:    802516 (osreldate)
> Hardware Platform:    amd64
> Processor Architecture:    amd64
>
> ZFS Storage pool Version:    28
> ZFS Filesystem Version:    5
>
> FreeBSD 8.2-STABLE #12: Thu Feb 9 11:35:23 EET 2012 root
> 10:53AM  up 56 mins, 6 users, load averages: 0.00, 0.00, 0.00
>
> ------------------------------------------------------------------------
>
>
>
>
> Background:
> we are using some tool that does indexing of some data and then pushes it
> into  database (currently bdb-5.2). Instances of indexer are running
> continuously one after another. Time of indexing for one instance of
> indexer may vary between  2 seconds and 30 minutes. But mostly it is
> below one minute. There is nothing else running on the machine except
> system stuff and daemons. After several hours of indexing i can see a lot
> of  active memory, it's ok. Then i check the number of vnodes. and it's
> really huge: 300k+ even tho nobody has so many opened files. Reading docs
> and googling i figured that's because of cached pages that reside in
> memory (unmounting of disk causes whole memory to be freed).  Also I
> figured that happens only when I am accessing files via mmap().
>
> Looks like pretty legit behaviour but the issue is:
> This spectacle continues (approximately for 12 hours) unlit indexers
> began to be killed out of swap. As I wrote above I observe a lot of used
> vnodes and like 7GB of active memory. I made a tool that allocates memory
> using malloc() to check what's the limit of available memory that can be
> allocated. It is several megabytes, sometimes more. Unless that tool gets
> killed out of swap as well. So how i can see the issue: for some reason
> after some process had exited normally all mapped pages don't get freed.
> I red about and I agree  that this is reasonable behaviour if we have
> spare memory. But following this logic these pages can be flushed back to
> file at any time when system is under stress conditions. So when I ask
> for a piece of RAM, OS should do that trick and give me what I ask. But
> that's never happens. Those pages are like frozen. Until I unmount disk.
> Even after there is not a single instance of indexer running.
>
> I believe all this is caused by mmap() for sure : BDB uses mmap() for
> accessing databases and we tested indexing with out pushing data to DB.
> Worked shiny. You may suggest that that's something wrong with BDB. But
> we have some more tools of ours that using mmap() as well and the
> behaviour is exact.
>
> Thank you. Paul, Ukraine.
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"

Hi Paul,

Are you using dedup anywhere on that pool?

Also, could you please post the full zfs-stats -a

-- 
George Kontostanos
Aicom telecoms ltd
http://www.aisecure.net