[REPORT] Some panics on ZFS

[Norikatsu Shigemura]

On Sat, 23 Dec 2006 15:23:32 +0100
Pawel Jakub Dawidek <pjd at freebsd.org> wrote:
> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> > ZFS filesystem version 3
> > ZFS storage pool version 3
> > Warning: memory type solaris leaked memory on destroy (116 allocations, 7328 bytes leaked).
> > ZFS filesystem version 3
> > ZFS storage pool version 3
> >  Warning: memory type solaris leaked memory on destroy (116 allocations, 7328 bytes leaked).
> > ZFS filesystem version 3
> > ZFS storage pool version 3
> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> This one is already fixed.

	Thank you.

> > 	But 'kldload zfs; mount -t zfs tank /mnt' didn't panic.  f(?_?)?
> It can be easly fixed, but bascially in zfs world you should mount file

	I saw perforce/112119.  Thank you.
	(http://perforce.freebsd.org/chv.cgi?CH=112119)

> systems after loading zfs.ko via:
> 	# zfs mount -a
> and activate zvols:
> 	# zfs volinit

	Yes.  So I'm using following configuration:-).

/etc/rc.conf.d/mountlate - - - - - - - - - - - - - - - - - - - - - - -
zfsinit () {
        kldstat -q -m zfs || kldload zfs
}

zfsmount () {
        zfs mount -a
        zfs volinit
}

zfsunmount () {
        zfs unmount -a
}

start_precmd=zfsinit
start_postcmd=zfsmount
stop_precmd=zfsunmount
/etc/rc.conf.d/mountlate - - - - - - - - - - - - - - - - - - - - - - -

	I think that /etc/rc.d/mountzfs will be installed.


> > 	3. When I'm testing with iozone, ZFS panic-ed.

> > 	I saw some memory modified after free.

> Are you able to setup memguard(9)? Or maybe you can give me procedure to
> reproduce it?

	I did setup memguard(9).  But I cannot get any data.  Because
	I got following panics:-).

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
panic: MEMGUARD: You must increase MAX_PAGES_PER_ITEM in memguard.c (requested: 512 pages)
cpuid = 0
KDB: enter: panic
[thread pid 817 tid 100070 ]
Stopped at      kdb_enter+0x30: leave
db> bt
Tracing pid 817 tid 100070 td 0x8e80a360
kdb_enter(806b6baf,0,806d113f,f9ef6a8c,8e80a360,...) at kdb_enter+0x30
panic(806d113f,200,0,f9ef6aa4,200000,...) at panic+0x14e
memguard_alloc(200000,1,8ebd359f,200000,20000,...) at memguard_alloc+0x31
malloc(200000,8ec4cb20,1,f9ef6af8,8ebd349e,...) at malloc+0xaf
kmem_alloc(200000,1,80000,0,f9ef6b38,...) at kmem_alloc+0x21
kmem_zalloc(200000,1,0,0,100,...) at kmem_zalloc+0x1e
dbuf_init(f9ef6b5c,8ec09cc7,8ec56638,8ec08950,1c,...) at dbuf_init+0x81
dmu_init(8ec56638,8ec08950,1c,8,8ebacc80,...) at dmu_init+0x8
spa_init(3,0,0,0,180,...) at spa_init+0x97
zfs_modevent(8ebacc80,0,0,75,f9ef6ba4,...) at zfs_modevent+0x58
module_register_init(8ec4da30,0,806b44ad,dc,0,...) at module_register_init+0xa5
linker_file_sysinit(8eb8a500,8ebad180,f9ef6bec,175,8ebad180,...) at linker_file_sysinit+0xe0
linker_load_file(8ebad180,f9ef6c14,0,72c,d0,...) at linker_load_file+0x125
linker_load_module(0,8e04c400,0,0,f9ef6c40,...) at linker_load_module+0xf9
kern_kldload(8e80a360,8e04c400,f9ef6c68,0,2f8,...) at kern_kldload+0xc4
kldload(8e80a360,f9ef6d00,4,c,f9ef6d38,...) at kldload+0x74
syscall(f9ef6d38) at syscall+0x2e3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x2, esp = 0x213, ebp = 0x7fbfeee4 ---
db> 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


	I increased MAX_PAGES_PER_ITEM to 65536.  But...
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
panic: MEMGUARD: Failed with M_WAITOK: memguard_map too small
cpuid = 0
KDB: enter: panic
[thread pid 1027 tid 100126 ]
Stopped at      kdb_enter+0x30: leave
db> bt
Tracing pid 1027 tid 100126 td 0x8ecd9d80
kdb_enter(806b6b8f,0,806d1172,f9fe3830,8ecd9d80,...) at kdb_enter+0x30
panic(806d1172,0,806d104e,f3,240,...) at panic+0x14e
memguard_alloc(240,2,0,240,8ecd8d80,...) at memguard_alloc+0x156
malloc(240,8ec50b20,2,f9fe389c,8ebd749e,...) at malloc+0xaf
kmem_alloc(240,2,84607000,9ee80400,f9fe38e4,...) at kmem_alloc+0x21
kmem_zalloc(240,2,8db77000,8db78000,7,...) at kmem_zalloc+0x1e
zio_create(25b9,0,9ee80400,aaf2b000,4000,...) at zio_create+0x2d
zio_write(8d416000,84607000,6,2,1,...) at zio_write+0x77
arc_write(8d416000,84607000,6,2,1,...) at arc_write+0x105
dbuf_sync(aafb2a50,8d416000,87109000,0,8ec48bf8,...) at dbuf_sync+0x3ca
dnode_sync(906d68f0,0,8d416000,87109000,8d416000,...) at dnode_sync+0x51b
dmu_objset_sync_dnodes(87109000,400,8ec48b11,806b5ba7,ae,...) at dmu_objset_sync_dnodes+0x97
dmu_objset_sync(85abc000,87109000,25b9,0,f9fe3bac,...) at dmu_objset_sync+0x60
dsl_dataset_sync(85aba000,87109000,0,84f8e1cc,84f8e184,...) at dsl_dataset_sync+0x24
dsl_pool_sync(84f8e000,25b9,0,1,0,...) at dsl_pool_sync+0x8f
spa_sync(84607000,25b9,0,84f8e15c,84f8e154,...) at spa_sync+0x453
txg_sync_thread(84f8e000,f9fe3d38,806b3a8a,328,8ecd8d80,...) at txg_sync_thread+0x1d7
fork_exit(8ec0fd20,84f8e000,f9fe3d38) at fork_exit+0xd1
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xf9fe3d6c, ebp = 0 ---
db> 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


> I'll try to reproduce iozone panic, I'd also like to reproduce "Memory
> modified after free" and "mi_switch: switch in a critical section"
> panics, can you tell me how to do it?

	All cases are using iozone.

	1. zpool create tank da0
	2. cd /tank; iozone -a |& tee iozone.1.log
	3. (panic) get backtrace and reset
	4. cd /tank; iozone -a |& tee iozone.2.log
	5. (panic) get backtrace and reset
	6. cd /tank; ......
		:

	And I got many type of panic messages.  So I don't know how to
	reproduce these behaviors.