ZFS kmem_map too small.

[Borja Marcos]

On Nov 6, 2007, at 11:00 AM, Pawel Jakub Dawidek wrote:

> If you use vm_kern.c.2.patch, can you show loader.conf and exact  
> command
> that can provke the panic?

I can elaborate a bit. I repeated the test while I kept some  
connections open,
watching zpool iostat, the Bonnie++ command output, the make  
buildworld, and
pinging the machine continuously.

Wired memory reached 1696 MB according to top, and the programs seemed  
to be stopped. I
didn't see more activity from top, zpool iostat, or make buildworld.  
But the kernel
was still answering to pings, although with a very long delay, an  
average of 300 - 500 ms.
I'm connected to the same local network, and the typical delay I get  
with the machine is
around 0.3 ms. It was until things started to go wrong.

The machine kept answering pings (very late, but with no packet loss),  
until I pressed
ctrl-C at the terminal session where I was running Bonnie++. That  
seemed to trigger the
panic, kmem_map to small.

And I've found something. I was using SCHED_ULE instead of SCHED_4BSD.  
I've switched back
to SCHED_4BSD (I apologise I didn't mention it, I thought it shouldn't  
make a difference) and
the kmem usage climbs much slower, sustaning a 30 MB/s write  
throughput during the Bonnie's "writing intelligently" test.

Using ULE, wired memory climbs and tops 1.6 GB in less than a minute,  
while using 4BSD it's
climbing much slower. I've seen it reach 1.2 GB, go back to 800 MB...  
and suddenly go down to
256 MB, while it keeps sustaining a write throughput of about 25 - 30  
MB/s (according to
zpool iostat at 10 second intervals).

I've compiled the kernel with debugger support and this is what the  
backtrace reads:

panic kmem_malloc(131072): kmem_map too small: 1608417280 total  
allocated cpuid=1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x17a
kmem_malloc at kmem_malloc+0x65e
uma_large_malloc() at uma_large_malloc+0x4a
malloc() at malloc+0x7d
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x147
vdev_queue_io_done() at vdev_queue_io_done+0x9c
vdev_geom_io_done() at vdev_geom_io_done+0x11
taskq_thread() at taskq_thread+0x17b
fork_exit() at fork_exit+0x11f
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xfffffffffee34d30, rbp = 0 ---
KDB: enter: panic



Hope it makes sense. Anything else I can provide?






Borja

----------------
"The thing he realised about the windows was this: because they had  
been converted into openable windows after they had first been  
designed to be impregnable, they were, in fact, much less secure than  
if they had been designed as openable windows in the first place."
    Douglas Adams, "Mostly Harmless"