ZFS "stalls" -- and maybe we should be talking about defaults?

[Daniel Kalchev]

On Mar 5, 2013, at 8:17 PM, Freddie Cash <fjwcash at gmail.com> wrote:

> 
> ZFS send/recv would eventually complete, but what used to take 15-20
> minutes would take 6-8 hours to complete.
> 
> I've reduced the ARC to only 32 GB, with arc_meta set to 28 GB, and things
> are running much smoother now (50-200 MB/s writes for 3-5 seconds every
> 10s), and send/recv is back down to 10-15 minutes.
> 
> Who would have thought "too much RAM" would be an issue?
> 
> Will play with this over the next couple of days with different ARC max
> settings to see where the problems start.  All of our ZFS boxes until this
> one had under 64 GB of RAM.  (And we had issues with dedupe enabled on
> boxes with too little RAM, as in under 32 GB.)

I have an archive box running very similar setup as yours, but with 72GB of RAM. I have set both arc_max and arc_meta_limit to 64GB, with no issues. I am still doing a very complex snapshot reordering between two pools. One of the pools has dedup enabled (which prompted me to add RAM), with dedup ratio f over 10x and there are still no issues or any stalling. The other pool has both dedup and compression for some filesystems. 

My only issue is that replacing a drive in either pool takes few days (6-drive vdevs of 3TB drives).

Perhaps the memory indexing/search algorithms are inefficient?

Daniel