8.1-RELEASE: [zfs] [kmem] zfs destroy snapshot results in panic: kmem_map too small

Leroy van Logchem leroy.vanlogchem at gmail.com
Fri Sep 24 11:52:50 UTC 2010

Problem          : Kernel panic "kmem_malloc(114688): kmem_map too small
Trigger          : Destroy ZFS snapshots (each bigger >80GB)
Version          : FreeBSD 8.1-RELEASE (GENERIC AMD64 but with DDB)
Workaround       : kmem / zfs arc tuning in /boot/loader.conf
System Info      : Manufacturer: Supermicro
                   Mainboard   : X8DTN
                   Cpu(s) Two  : Intel(R) Xeon(R) E5520  @ 2.27GHz
                   Memory      : 10240 MB (speed: 1066 MHz)

Storage          : mpt0 Adapter:
                   Board Name: SAS3081E
                   Board Assembly: L3-00159-02D
                   Chip Name: C1068E

24 drives are connected to mpt0: /dev/da[0 up to 23]

Each drive       :
                   512             # sectorsize
                   2000398934016   # mediasize in bytes (1.8T)
                   3907029168      # mediasize in sectors
                   243201          # Cylinders according to firmware.
                   255             # Heads according to firmware.
                   63              # Sectors according to firmware.
                   JK1130YAHUSJMT    # Disk ident.

# zpool list
tank  43.5T  31.0T  12.5T    71%  ONLINE  -

This server has been running since dec 2009 without downtime.
We keep 14 daily zfs snapshots. The problem of our nightly kernel panic started
when the size of each snapshot exceeded about 80 GB. A cronjob first creates
new snapshots just fine and destroys expired (older >14 days) snapshots.

During the command "zfs destroy <snapname>" the memory free seen in top(1)
drops very quickly and soon after that this panic is generated:

panic: kmem_malloc(114688): kmem_map too small: 3307884544 total allocated
cpuid = 2
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x182
kmem_malloc() at kmem_malloc+0x5b5
uma_large_malloc() at uma_large_malloc+0x4a
malloc() at malloc+0x14b
zio_compress_data() at zio_compress_data+0xa2
zio_write_bp_init() at zio_write_bp_init+0xc2
zio_exectute() at zio_execute+0x77
taskq_run_safe() at taskq_run_safe+0x13
taskqueue_run() at taskqueue_run+0x91
taskqueue_thread_loop() at taskqueue_thread_loop+0x3f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff81261edd30, rbp = 0 ---
panic: kmem_malloc(118784): kmem_map too small: 3307884544 total allocated
cpuid = 2

During these panics no tuning or /boot/loader.conf values where present.

Only after hitting this behaviour yesterday i created boot/loader.conf:


After which the snapshots could be destroyed without panic.


Are users required to tune?
If so, are there reliable guidelines regarding memory and ZFS?
Are the -quickly- chosen values any good? (service window was closing)



More information about the freebsd-stable mailing list