kernel panic while writing to a ZFS volume on iSCSI LUN

[Ulf Lilleengen]

On Mon, Feb 23, 2009 at 05:00:06PM +0100, ml-ktk at netlabs.org wrote:
> Hi,
> 
> I am testing ZFS, iSCSI & snapshots on FreeBSD as I want to use it later
> for some productive data. While doing an rsync from a maildir with lots
> of small files, the box crashed. Until then there were about 3.5GB of
> data transferred
> 
> Platform: FreeBSD 7.1, i386, install from CD
> Box: Xeon 3.2 something
> RAM: 1.5GB
> 50GB iSCSI LUN on Network Appliance Filer
> 
> I tried to get some hints out of the dump file according to
> 
> http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.html
> 
> but I noticed I had to get source first & compile the kernel on my own.
> The backtrace here is done using a kernel.debug which got compiled out
> of RELENG_7_1, but it was *not* the same kernel which was running at the
> time of the crash (kernel was the default one, not the compiled one).
> Does that make a difference for my case or is that ok like this?
> 
> Message in /var/log/messages on reboot:
> 
> -
> Feb 20 00:01:23 chewbacca savecore: reboot after panic:
> kmem_malloc(131072): kmem_map too small: 228896768 total allocated
> Feb 20 00:01:23 chewbacca savecore: writing core to vmcore.0
> -
> 
> backtrace:
> http://freebsd.pastebin.com/m36e7ddd0
> 
> My question is if it is crashing in iSCSI or ZFS code & what I should
> try to get rid of it  :-)
> 
This is a problem with ZFS due to exhaustion of the kernel memory address
space (ZFS is quite hungry for memory). This can be solved by finely tuning
the different limits specified here:

http://wiki.freebsd.org/ZFSTuningGuide

You might have to do some try and fail to get it to work, but my experience
is that the problems is usually solveable if you invest enough time in the
tuning.

-- 
Ulf Lilleengen