ZFS on 10-STABLE r281159: programs, accessing ZFS pauses for minutes in state [*kmem arena]

Steven Hartland killing at multiplay.co.uk
Mon Nov 30 12:49:40 UTC 2015



On 31/07/2015 22:27, Don Lewis wrote:
> On 30 Jul, Konstantin Belousov wrote:
>> On Thu, Jul 30, 2015 at 02:30:08PM +0300, Lev Serebryakov wrote:
>>> Hello Freebsd-fs,
>>>
>>>
>>>    I'm migrating my NAS from geom_raid5 + UFS to ZFS raidz. My main storage
>>> is 5x2Tb HDDs. Additionaly, I have 2x3Tb HDDs attached to hold my data when
>>> I re-make my main storage.
>>>
>>>   So, I have now two ZFS pools:
>>>
>>> ztemp mirror ada0 ada1 [both are 3Tb HDDS]
>>> zstor raidz ada3 ada4 ada5 ada6 ada7 [all of them are 2Tb]
>>>
>>>   ztemp contain one filesystem with 2.1Tb of my data. ztemp was populated
>>>   with my data from old geom_raid5 + UFS installation via "rsync" and it was
>>>   FAST (HDD-speed).
>>>
>>>   zstor contains several empty file systems (one per user), like:
>>>
>>> zstor/home/lev
>>> zstor/home/sveta
>>> zstor/home/nsvn
>>> zstor/home/torrents
>>> zstor/home/storage
>>>
>>>   Deduplication IS TURNED OFF. atime is turned off. Record size set to 1M as
>>> I have a lot of big files (movies, RAW photo from DSLR, etc). Compression is
>>> turned off.
>>>
>>>   When I try to copy all my data from temporary HDDs (ztemp pool) to my new
>>> shiny RIAD (zstor pool) with
>>>
>>> cd /ztemp/fs && rsync -avH lev sveta nsvn storage /usr/home/
>>>
>>>   rsync pauses for tens of minutes (!) after several hundreds of files. ^T
>>> and top shows state "[*kmem arena]". When I stop rsync with ^C and try to do
>>> "zfs list" it waits forever, in state "[*kmem arena]" again.
>> Show the output of sysctl debug.vmem_check.
>>
>>>   This server is equipped with 6GiB of RAM.
>>>
>>>   It looks FreeBSD contains bug about year ago which leads to this behavior,
>>> but mailing lists says, that it was fixed in r272221, 10 months ago.
> I think I may have gotten bitten by this yesterday on a fairly recent
> 10.2-PRERELEASE machine with 8 GB of RAM.  It's nominally a zfs-only
> machine, but I had some data on a couple of UFS drives that I needed to
> copy over to a zfs filesystem.  I connected one of the drives to a sata
> to usb adapater and plugged it into the machine, then ran rsync to
> transfer the contents of a ~100 GB filesystem.  I had a number of active
> programs running, including a rather bloated firefox process that had
> gobbled lots of ram.  In my case, arc stayed small (< 1 GB), inactive
> memory was a couple of GB, and several GB of data got pushed to swap.
> Free memory got very low, bouncing around in the 10's of MB for a while
> before the machine locked.  It wasn't totally dead because my X11
> desktop is configured in focus follow mouse mode and I could see the
> window focus change when I moved the mouse around.  Eventually I did
> something to provoke the window manager and/or the Xorg server into
> locking up as well.  I wasn't able to switch to console mode.  I
> eventually gave up and hit the reset button.
>
> %sysctl debug.vmem_check
> sysctl: unknown oid 'debug.vmem_check': No such file or directory
>
> With the same set of processes running, but no UFS, this is what top
> says about memory usage:
>
> Mem: 1156M Active, 3403M Inact, 1682M Wired, 31M Cache, 1631M Free
> ARC: 1129M Total, 588M MFU, 492M MRU, 54K Anon, 10M Header, 39M Other
> Swap: 40G Total, 40G Free
>
This is a little late but just wanted to note for the record that the 
combination of r281026, r281108, r281109 (MFC10: r282361) and r282690 
(MFC10: r283310) should address this issue, as noted on the PR: 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194513

If anyone still experiences this please let us know on the PR.

     Regards
     Steve


More information about the freebsd-fs mailing list