kern/118487: zfs related kernel panic on RELENG_7 (7.0BETA4):
kmem_malloc(XXX): kmem_map too small
Michael Haro
mharo at FreeBSD.org
Fri Dec 7 13:30:04 PST 2007
>Number: 118487
>Category: kern
>Synopsis: zfs related kernel panic on RELENG_7 (7.0BETA4): kmem_malloc(XXX): kmem_map too small
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Fri Dec 07 21:30:04 UTC 2007
>Closed-Date:
>Last-Modified:
>Originator: Michael Haro
>Release: FreeBSD 7.0-BETA4 i386
>Organization:
>Environment:
System: FreeBSD zfsserver.mtv.bitsurf.net 7.0-BETA4 FreeBSD 7.0-BETA4 #2: Thu Dec 6 23:02:30 PST 2007 root at zfsserver.mtv.bitsurf.net:/usr/obj/usr/src/sys/KERNEL i386
I have 3 kernel dumps. I suspect they're related...
Unread portion of the kernel message buffer:
panic: kmem_malloc(131072): kmem_map too small: 418766848 total allocated
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper(c09337fe,f3155af0,c063d17f,c0951d54,0,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c0951d54,0,c094a215,f3155afc,0,...) at kdb_backtrace+0x29
panic(c094a215,20000,18f5e000,c094a20f,7d0,...) at panic+0x10f
kmem_malloc(c147108c,20000,2,f3155b7c,c08541b0,...) at kmem_malloc+0x284
page_alloc(0,20000,f3155b6f,2,24539a8,...) at page_alloc+0x27
uma_large_malloc(20000,2,c683ccf0,c6488ac8,28000,...) at uma_large_malloc+0x50
malloc(20000,c0b9f1c0,2,f3155bc0,c0b7a719,...) at malloc+0x8d
zfs_kmem_alloc(20000,2,f3155c48,c0b699bc,20000,...) at zfs_kmem_alloc+0x20
zio_buf_alloc(20000,c6488ac8,1,3b7,33860000,...) at zio_buf_alloc+0x19
vdev_queue_io_to_issue(f3155c64,c6bcf228,c0b9c2d0,133,c43ab2e4,...) at vdev_queue_io_to_issue+0x1bc
vdev_queue_io_done(c6bcf228,f3155ce0,f3155ce0,f3155cf4,c0b23ba0,...) at vdev_queue_io_done+0xa8
vdev_geom_io_done(c6bcf228,0,c0b976f3,33d,2f652a7e,...) at vdev_geom_io_done+0x12
taskq_thread(c3f9a7f8,f3155d38,4,0,85,...) at taskq_thread+0x140
fork_exit(c0b23a60,c3f9a7f8,f3155d38) at fork_exit+0x97
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xf3155d70, ebp = 0 ---
Uptime: 9m48s
Physical memory: 1011 MB
Dumping 473 MB: 458 442 426 410 394 378 362 346 330 314 298 282 266 250 234 218 202 186 170 154 138 122 106 90 74 58 42 26 10
Unread portion of the kernel message buffer:
ad16: FAILURE - device detached
subdisk16: detached
ad16: detached
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x2c
fault code = supervisor write, page not present
instruction pointer = 0x20:0xc0631085
stack pointer = 0x28:0xef33bc5c
frame pointer = 0x28:0xef33bc70
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 3 (g_up)
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper(c09337fe,ef33baf8,c063d17f,c0951d54,0,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c0951d54,0,c0913383,ef33bb04,0,...) at kdb_backtrace+0x29
panic(c0913383,c0952ffb,c3f544d0,1,1,...) at panic+0x10f
trap_fatal(c0a12600,0,2,8,76c5ec12,...) at trap_fatal+0x333
trap_pfault(c0a120a8,ef33bb90,c066ac9d,ef33bbb4,c,...) at trap_pfault+0x250
trap(ef33bc1c) at trap+0x3c6
calltrap() at calltrap+0x6
--- trap 0xc, eip = 0xc0631085, esp = 0xef33bc5c, ebp = 0xef33bc70 ---
_mtx_lock_flags(1c,0,c0b9de0d,1d8,c0b932b0,...) at _mtx_lock_flags+0x15
vdev_geom_io_intr(c431c948,c09c6304,0,0,0) at vdev_geom_io_intr+0x44
biodone(c431c948,c0a12008,24c,c092bdb9,64,...) at biodone+0xad
g_io_schedule_up(c3f0dc60,4c,c092ca83,5b,0,...) at g_io_schedule_up+0x7f
g_up_procbody(0,ef33bd38,102640,ffffefff,ffffffff,...) at g_up_procbody+0x6c
fork_exit(c05ed200,0,ef33bd38) at fork_exit+0x97
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xef33bd70, ebp = 0 ---
Uptime: 11m42s
Physical memory: 1011 MB
Dumping 340 MB: 325 309 293 277 261 245 229 213 197 181 165 149 133 117 101 85 69 53 37 21 5
Unread portion of the kernel message buffer:
panic: kmem_malloc(32768): kmem_map too small: 533876736 total allocated
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper(c09337fe,f320c034,c063d17f,c0951d54,0,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c0951d54,0,c094a215,f320c040,0,...) at kdb_backtrace+0x29
panic(c094a215,8000,1fd25000,c094a20f,7d0,...) at panic+0x10f
kmem_malloc(c147108c,8000,2,f320c0c0,c08541b0,...) at kmem_malloc+0x284
page_alloc(0,8000,f320c0b3,2,2eed800,...) at page_alloc+0x27
uma_large_malloc(8000,2,400,c44d8000,0,...) at uma_large_malloc+0x50
malloc(8000,c0b9f1c0,2,f320c104,c0b7a719,...) at malloc+0x8d
zfs_kmem_alloc(8000,2,f320c1f8,c0b6b570,8000,...) at zfs_kmem_alloc+0x20
zio_buf_alloc(8000,0,400,0,f320c13c,...) at zio_buf_alloc+0x19
vdev_raidz_io_start(c67ab8a0,c859ee00,c43a9400,efeb8000,95,...) at vdev_raidz_io_start+0x390
vdev_mirror_io_start(c67abac8,20000,0,c859ee00,1,...) at vdev_mirror_io_start+0x1fc
zio_dva_allocate(c67abac8,24a24,0,c859ee00,c61679c4,...) at zio_dva_allocate+0x115
zio_write_compress(c67abac8,c0b9cbac,329,326,1,...) at zio_write_compress+0x178
zio_wait_for_children(c67abac8,c6cb1900,7dd,760,c44dc9b4,...) at zio_wait_for_children+0x63
dbuf_sync_list(c8515c38,c6cb1900,c0b97eb7,74a,c4509cb0,...) at dbuf_sync_list+0x327
dbuf_sync_list(c67c0c38,c6cb1900,c0b97eb7,74a,c4509cb0,...) at dbuf_sync_list+0x117
dbuf_sync_list(c4509d50,c6cb1900,257,250,0,...) at dbuf_sync_list+0x117
dnode_sync(c4509cb0,c6cb1900,0,a0,c4fd2800,...) at dnode_sync+0x1e2
dmu_objset_sync_dnodes(c510bae0,c6cb1900,7,3,2,...) at dmu_objset_sync_dnodes+0x37
dmu_objset_sync(c4fd2800,c8e5f678,c6cb1900,0,c4e8d898,...) at dmu_objset_sync+0x15a
dsl_pool_sync(c4e8d800,24a24,0,0,0,...) at dsl_pool_sync+0x98
spa_sync(c41bc000,24a24,0,13a,c4e8d940,...) at spa_sync+0x3f7
txg_sync_thread(c4e8d800,f320cd38,fe36743f,21a9e16c,ac45c4ff,...) at txg_sync_thread+0x1ea
fork_exit(c0b61b90,c4e8d800,f320cd38) at fork_exit+0x97
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xf320cd70, ebp = 0 ---
Uptime: 17m48s
Physical memory: 1011 MB
Dumping 559 MB: 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16
dmesg:
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-BETA4 #2: Thu Dec 6 23:02:30 PST 2007
root at zfsserver.mtv.bitsurf.net:/usr/obj/usr/src/sys/KERNEL
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) XP 2000+ (1659.61-MHz 686-class CPU)
Origin = "AuthenticAMD" Id = 0x680 Stepping = 0
Features=0x383f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
AMD Features=0xc0400800<SYSCALL,MMX+,3DNow!+,3DNow!>
real memory = 1073676288 (1023 MB)
avail memory = 1036881920 (988 MB)
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <AMIINT SiS735XX> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <SiS 735 host to AGP bridge> on hostb0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> mem 0xce000000-0xceffffff,0xb0000000-0xb7ffffff,0xbdb80000-0xbdbfffff irq 11 at device 0.0 on pci1
isab0: <PCI-ISA bridge> at device 2.0 on pci0
isa0: <ISA bus> on isab0
ohci0: <SiS 5571 USB controller> mem 0xcfffd000-0xcfffdfff irq 10 at device 2.2 on pci0
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb0: OHCI version 1.0, legacy support
usb0: <SiS 5571 USB controller> on ohci0
usb0: USB revision 1.0
uhub0: <SiS OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 3 ports with 3 removable, self powered
ohci1: <SiS 5571 USB controller> mem 0xcfffe000-0xcfffefff irq 12 at device 2.3 on pci0
ohci1: [GIANT-LOCKED]
ohci1: [ITHREAD]
usb1: OHCI version 1.0, legacy support
usb1: <SiS 5571 USB controller> on ohci1
usb1: USB revision 1.0
uhub1: <SiS OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 3 ports with 3 removable, self powered
atapci0: <SiS 735 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 2.5 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <multimedia, audio> at device 2.7 (no driver attached)
sis0: <SiS 900 10/100BaseTX> port 0xc800-0xc8ff mem 0xcfffc000-0xcfffcfff irq 10 at device 3.0 on pci0
miibus0: <MII bus> on sis0
rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus0
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sis0: Ethernet address: 00:0a:e6:8a:3b:e3
sis0: [ITHREAD]
pci0: <multimedia, video> at device 11.0 (no driver attached)
pci0: <multimedia, video> at device 13.0 (no driver attached)
atapci1: <Promise PDC20375 SATA150 controller> port 0xdc00-0xdc3f,0xd800-0xd80f,0xd400-0xd47f mem 0xcffff000-0xcfffffff,0xcffc0000-0xcffdffff irq 10 at device 15.0 on pci0
atapci1: [ITHREAD]
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci1
ata4: [ITHREAD]
atapci2: <Promise PDC40518 SATA150 controller> port 0xc400-0xc47f,0xc000-0xc0ff mem 0xcfffb000-0xcfffbfff,0xcff80000-0xcff9ffff irq 11 at device 17.0 on pci0
atapci2: [ITHREAD]
atapci2: [ITHREAD]
ata5: <ATA channel 0> on atapci2
ata5: [ITHREAD]
ata6: <ATA channel 1> on atapci2
ata6: [ITHREAD]
ata7: <ATA channel 2> on atapci2
ata7: [ITHREAD]
ata8: <ATA channel 3> on atapci2
ata8: [ITHREAD]
uhci0: <VIA 83C572 USB controller> port 0xb800-0xb81f irq 11 at device 19.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb2: <VIA 83C572 USB controller> on uhci0
usb2: USB revision 1.0
uhub2: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xbc00-0xbc1f irq 11 at device 19.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb3: <VIA 83C572 USB controller> on uhci1
usb3: USB revision 1.0
uhub3: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xcfffaf00-0xcfffafff irq 5 at device 19.2 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 0.95
usb4: companion controllers, 2 ports each: usb2 usb3
usb4: <VIA VT6202 USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 4 ports with 4 removable, self powered
acpi_button0: <Sleep Button> on acpi0
fdc0: <floppy drive controller> port 0x3f2-0x3f3,0x3f4-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
pmtimer0 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
uhub5: <Dell Dell USB Keyboard Hub, class 9/0, rev 1.10/2.00, addr 2> on uhub2
uhub5: 3 ports with 2 removable, bus powered
ukbd0: <Dell Dell USB Keyboard Hub, class 0/0, rev 1.10/2.00, addr 3> on uhub5
kbd2 at ukbd0
uhid0: <Dell Dell USB Keyboard Hub, class 0/0, rev 1.10/2.00, addr 3> on uhub5
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
Timecounter "TSC" frequency 1659613591 Hz quality 800
Timecounters tick every 1.000 msec
ad0: 114473MB <WDC WD1200BB-00CJA1 17.07W17> at ata0-master UDMA100
ZFS filesystem version 6
ZFS storage pool version 6
acd0: DVDROM <HL-DT-STDVD-ROM GDR8163B/0L23> at ata1-slave UDMA33
ad8: 239372MB <Maxtor 6Y250P0 YAR41BW0> at ata4-master UDMA133
ad10: 238475MB <HDS722525VLSA80 V36OA6MA> at ata5-master SATA150
ad12: 238475MB <HDS722525VLSA80 V36OA6MA> at ata6-master SATA150
ad14: 238475MB <HDS722525VLSA80 V36OA6MA> at ata7-master SATA150
ad16: 238475MB <HDS722525VLSA80 V36OA6MA> at ata8-master SATA150
Trying to mount root from zfs:tank
Using the GENERIC kernel with DDB and KDB additions.
/boot/loader.conf:
autoboot_delay="2"
zfs_load="YES"
vfs.root.mountfrom="zfs:tank"
#vfs.zfs.prefetch_disable="1"
vm.kmem_size="512M"
vm.kmem_size_max="512M"
/usr/src was synced to RELENG_7 yesterday (and used to build the kernel that paniced).
zfsserver# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
data 1.14T 812G 352G 69% ONLINE -
tank 110G 71.5G 38.5G 64% ONLINE -
zfsserver# zfs list
NAME USED AVAIL REFER MOUNTPOINT
data 649G 267G 511G /data
data/backups 30.4K 267G 30.4K /data/backups
data/mythtv 138G 267G 138G /data/mythtv
tank 72.5G 35.8G 462M legacy
tank/backups 20.0G 35.8G 20.0G /tank/backups
tank/home 126K 35.8G 26.5K /home
tank/home/mharo 100K 35.8G 100K /home/mharo
tank/random 47.0G 35.8G 47.0G /tank/random
tank/swap 155K 36.8G 155K -
tank/usr 2.64G 35.8G 2.11G /usr
tank/usr/ports 392M 35.8G 139M /usr/ports
tank/usr/ports/distfiles 254M 35.8G 254M /usr/ports/distfiles
tank/usr/src 152M 35.8G 152M /usr/src
tank/var 1.43G 35.8G 1.43G /var
tank/var/log 414K 35.8G 414K /var/log
>Description:
After a few minutes of moving data from one zfs partition to another in the same
zpool the machine panics (see above).
I suspect this is zfs related, but don't know for sure.
>How-To-Repeat:
cd /data
mv -v mythtv.orig/recordings/*.nuv mythtv/recordings/
wait a few minutes
I think it happens when copying files > 1G in size, but I haven't traced it
down to which file was being moved when the system paniced.
I've tried setting vfs.zfs.prefetch_disable="1" and vfs.zfs.prefetch_disable="0" but
it doesn't seemt to matter.
I also tried applying http://people.freebsd.org/~pjd/patches/vm_kern.c.2.patch
I *think* that made my system more stable, but I'm still crashing in under 30 minutes.
>Fix:
Unknown. Willing to try other things.
>Release-Note:
>Audit-Trail:
>Unformatted:
More information about the freebsd-bugs
mailing list