Troubleshooting kernel panic with zfs

[Josh Gitlin]

Thanks for the help Mike! In reviewing our settings I also found that we're setting vm.kmem_size to 1.5x physical RAM installed, based on a tuning guide someone found. I personally believe that might be misguided and may be related to the panic we saw, as this causes zfs-stats -a to show the kernel memory map being larger than system RAM:

Kernel Memory Map:				24.00	GiB
	Size:				21.72%	5.21	GiB
	Free:				78.28%	18.79	GiB

I'll try tuning these settings down and see if the problem happens again. Appreciate the support!

--
 <http://www.goboomtown.com/>	
Josh Gitlin
Senior DevOps Engineer
(415) 690-1610 x155

Stay up to date and join the conversation in Relay <http://relay.goboomtown.com/>.

> On Sep 21, 2018, at 9:04 AM, Mike Tancsa <mike at sentex.net> wrote:
> 
> On 9/20/2018 8:27 PM, Josh Gitlin wrote:
>> 
>> My hunch is that, given this was inside kmem_malloc, we were unable to allocate memory for a zio_write_compress call (the pool does have ZFS compression on) and hence this is a tuning issue and not a bug... but I am looking for confirmation and/or suggested changes/troubleshooting steps. The ZFS tuning configuration has been stable for years, to it may be a change in behavior or traffic... If this looks like it might be a bug, I will be able to get more information from a minidump if it reoccurs and can follow up on this thread.
> 
> Given all the issues with memory pressure perhaps ARC is eating too
> much, I would try artificially limiting it so it does not eat up too
> much RAM.  We had a similar purposed server but with 32G RAM and I found
> one day the system was randomly shooting processes to reclaim RAM
> 
> eg
> pid xxx (yyy), uid zzzz, was killed: out of swap space
> 
> 
> Perhaps
> 
> sysctl -w vfs.zfs.arc_max=6000000000
> 
> eg.
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229764
> 
> https://lists.freebsd.org/pipermail/freebsd-questions/2018-July/282261.html
> 
> 	---Mike
> 
> 
> 
> 
> -- 
> -------------------
> Mike Tancsa, tel +1 519 651 3400 x203
> Sentex Communications, mike at sentex.net
> Providing Internet services since 1994 www.sentex.net
> Cambridge, Ontario Canada