Tuning ZFS ARC for 512GB Memory

[Reed A. Cartwright]

I have a 512GB memory system that has a 480GB (or so) L2ARC, and a
raidz2 storage pool.  No dedup or compression is enabled.  Although, I
will enable compression in the future.

By default vfs.zfs.arc_max is about 300GB and the L2ARC is hardly touched.

I've seen a couple hints recently on the mailing lists that having too
big of an ARC can be a performance hit because it takes a long time to
inspect it when something needs to integrate over the entries.

I've also had issues with running 9.1 on this machine due to zfs
deadlocks or stalls and am currently "stuck" using 9.0 with no issue.

So, I was wondering if shrinking the ARC would potentially help and
what should I shrink it to?

I am thinking about setting vfs.zfs.arc_max to 64GB.

Does anyone have an opinion?


--
Reed A. Cartwright, PhD
Assistant Professor of Genomics, Evolution, and Bioinformatics
School of Life Sciences
Center for Evolutionary Medicine and Informatics
The Biodesign Institute
Arizona State University