ZFS "stalls" -- and maybe we should be talking about defaults?
daniel at digsys.bg
Tue Mar 5 23:08:38 UTC 2013
On Mar 5, 2013, at 8:17 PM, Freddie Cash <fjwcash at gmail.com> wrote:
> ZFS send/recv would eventually complete, but what used to take 15-20
> minutes would take 6-8 hours to complete.
> I've reduced the ARC to only 32 GB, with arc_meta set to 28 GB, and things
> are running much smoother now (50-200 MB/s writes for 3-5 seconds every
> 10s), and send/recv is back down to 10-15 minutes.
> Who would have thought "too much RAM" would be an issue?
> Will play with this over the next couple of days with different ARC max
> settings to see where the problems start. All of our ZFS boxes until this
> one had under 64 GB of RAM. (And we had issues with dedupe enabled on
> boxes with too little RAM, as in under 32 GB.)
I have an archive box running very similar setup as yours, but with 72GB of RAM. I have set both arc_max and arc_meta_limit to 64GB, with no issues. I am still doing a very complex snapshot reordering between two pools. One of the pools has dedup enabled (which prompted me to add RAM), with dedup ratio f over 10x and there are still no issues or any stalling. The other pool has both dedup and compression for some filesystems.
My only issue is that replacing a drive in either pool takes few days (6-drive vdevs of 3TB drives).
Perhaps the memory indexing/search algorithms are inefficient?
More information about the freebsd-stable