kern/167109: zfs diff kernel panic Fatal trap 9: general protection fault

Claudius Herder claudius at ambtec.de
Thu Apr 19 19:30:11 UTC 2012


>Number:         167109
>Category:       kern
>Synopsis:       zfs diff kernel panic Fatal trap 9: general protection fault
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Apr 19 19:30:10 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     Claudius Herder
>Release:        FreeBSD 9.0-RELEASE
>Organization:
>Environment:
FreeBSD server.ambtec.de 9.0-RELEASE FreeBSD 9.0-RELEASE #2: Thu Apr 19 19:09:14 CEST 2012     claudius at server.ambtec.de:/usr/obj/usr/src/sys/CUSTOM  amd64
>Description:
I already posted to freebsd-fs at freebsd.org but got no response:

I received a kernel panic after running zfs diff on my system/rootfs filesystem, other filesystems work without problems, but i have not tested them all.

Apr 15 21:11:17 server kernel: Fatal trap 9: general protection fault
while in kernel mode
Apr 15 21:11:17 server kernel: cpuid = 3; apic id = 03
Apr 15 21:11:17 server kernel: instruction pointer      =
0x20:0xffffffff80b0ff91
Apr 15 21:11:17 server kernel: stack pointer            =
0x28:0xffffff845fa1b430
Apr 15 21:11:17 server kernel: frame pointer            =
0x28:0xffffff845fa1b470
Apr 15 21:11:17 server kernel: code segment             = base 0x0,
limit 0xfffff, type 0x1b
Apr 15 21:11:17 server kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Apr 15 21:11:17 server kernel: processor eflags = interrupt enabled,
resume, IOPL = 0
Apr 15 21:11:17 server kernel: current process          = 1797 (zfs)
Apr 15 21:11:17 server kernel: trap number              = 9
Apr 15 21:11:17 server kernel: panic: general protection fault
Apr 15 21:11:17 server kernel: cpuid = 3
Apr 15 21:11:17 server kernel: KDB: stack backtrace:
Apr 15 21:11:17 server kernel: #0 0xffffffff803fcb5e at kdb_backtrace+0x5e
Apr 15 21:11:17 server kernel: #1 0xffffffff803c9ce3 at panic+0x183
Apr 15 21:11:17 server kernel: #2 0xffffffff805c49a0 at trap_fatal+0x290
Apr 15 21:11:17 server kernel: #3 0xffffffff805c4eda at trap+0x10a
Apr 15 21:11:17 server kernel: #4 0xffffffff805af7ef at calltrap+0x8
Apr 15 21:11:17 server kernel: #5 0xffffffff80b0f0e0 at
fzap_cursor_retrieve+0x110
Apr 15 21:11:17 server kernel: #6 0xffffffff80b13bc5 at
zap_cursor_retrieve+0x155
Apr 15 21:11:17 server kernel: #7 0xffffffff80b0e12f at
zap_value_search+0x7f
Apr 15 21:11:17 server kernel: #8 0xffffffff80b18912 at
zfs_obj_to_path_impl+0x292
Apr 15 21:11:17 server kernel: #9 0xffffffff80b18b45 at
zfs_obj_to_stats+0x175
Apr 15 21:11:17 server kernel: #10 0xffffffff80b3075e at
zfs_ioc_obj_to_stats+0x7e
Apr 15 21:11:17 server kernel: #11 0xffffffff80b33a46 at zfsdev_ioctl+0xe6
Apr 15 21:11:17 server kernel: #12 0xffffffff8034e4db at devfs_ioctl_f+0x7b
Apr 15 21:11:17 server kernel: #13 0xffffffff8040e185 at kern_ioctl+0x115
Apr 15 21:11:17 server kernel: #14 0xffffffff8040e3b0 at sys_ioctl+0xf0
Apr 15 21:11:17 server kernel: #15 0xffffffff805c4290 at amd64_syscall+0x450
Apr 15 21:11:17 server kernel: #16 0xffffffff805afad7 at Xfast_syscall+0xf7

I tried different snapshots on rootfs to narrow down then problem, but
after a few forced reboots and zpool scrub after each crash zfs diff
hangs only.

procstat -k -k 8428
 PID    TID COMM             TDNAME           KSTACK
 8428 100719 zfs              -                mi_switch+0x174
sleepq_catch_signals+0x2f4 sleepq_wait_sig+0xc _sleep+0x279
pipe_write+0x125e write_record+0x79 report_free_dnode_range+0x2c
diff_cb+0x1bf traverse_visitbp+0x21c traverse_visitbp+0x309
traverse_visitbp+0x309 traverse_visitbp+0x309 traverse_visitbp+0x309
traverse_visitbp+0x309 traverse_visitbp+0x309 traverse_dnode+0x7c
traverse_visitbp+0x47f traverse_impl+0x188
 8428 101237 zfs              -                mi_switch+0x174
sleepq_wait+0x42 _sx_slock_hard+0x1f2 _sx_slock+0x45
zap_get_leaf_byblk+0xbd zap_deref_leaf+0x68 fzap_cursor_retrieve+0xe7
zap_cursor_retrieve+0x155 zap_value_search+0x7f
zfs_obj_to_path_impl+0x292 zfs_obj_to_stats+0x175
zfs_ioc_obj_to_stats+0x7e zfsdev_ioctl+0xe6 devfs_ioctl_f+0x7b
kern_ioctl+0x115 sys_ioctl+0xf0 amd64_syscall+0x450 Xfast_syscall+0xf7

 zdb:
system:
    version: 28
    name: 'system'
    state: 0
    txg: 744
    pool_guid: 10799670079652932983
    hostid: 3191383858
    hostname: 'tmpserver.ambtec.de'
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 10799670079652932983
        children[0]:
            type: 'mirror'
            id: 0
            guid: 2010203102961720602
            metaslab_array: 30
            metaslab_shift: 34
            ashift: 12
            asize: 2966227255296
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 4310542124404815932
                path: '/dev/gpt/system0'
                phys_path: '/dev/gpt/system0'
                whole_disk: 1
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 14038163402933743119
                path: '/dev/gpt/system1'
                phys_path: '/dev/gpt/system1'
                whole_disk: 1
                create_txg: 4


dmesg:

Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-RELEASE #2: Thu Apr 19 19:09:14 CEST 2012
    claudius at server.ambtec.de:/usr/obj/usr/src/sys/CUSTOM amd64
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (3411.56-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206a7  Family = 6  Model = 2a  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x17bae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,AVX>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16436305920 (15674 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
acpi0: <ALASKA A M I> on motherboard
ACPI Error: [RAMB] Namespace lookup failure, AE_NOT_FOUND (20110527/psargs-392)
ACPI Exception: AE_NOT_FOUND, Could not execute arguments for [RAMW] (Region) (20110527/nsinit-380)
acpi0: Power Button (fixed)
acpi0: reservation of 67, 1 (4) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xf7800000-0xf7bfffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
pci0: <simple comms> at device 22.0 (no driver attached)
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xf7d04000-0xf7d043ff irq 23 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0: <EHCI (generic) USB 2.0 controller> on ehci0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci3: <ACPI PCI bus> on pcib3
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xf7c00000-0xf7c07fff irq 17 at device 0.0 on pci3
xhci0: 32 byte context size.
usbus1 on xhci0
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci4: <ACPI PCI bus> on pcib4
re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xf0004000-0xf0004fff,0xf0000000-0xf0003fff irq 18 at device 0.0 on pci4
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 54:04:a6:7e:eb:34
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 19 at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib6
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xf7d03000-0xf7d033ff irq 23 at device 29.0 on pci0
usbus2: EHCI version 1.0
usbus2: <EHCI (generic) USB 2.0 controller> on ehci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Cougar Point AHCI SATA controller> port 0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xf7d02000-0xf7d027ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
ZFS filesystem version 5
ZFS storage pool version 28
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
usbus1: 5.0Gbps Super Speed USB v3.0
usbus2: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <0x1b21> at usbus1
uhub1: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ST3000DM001-9YN166 CC47> ATA-8 SATA 3.x device
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <ST3000DM001-9YN166 CC47> ATA-8 SATA 3.x device
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #7 Launched!
Timecounter "TSC-low" frequency 13326397 Hz quality 1000
GEOM_MIRROR: Device mirror/swap launched (2/2).
uhub1: 4 ports with 4 removable, self powered
Trying to mount root from zfs:system/rootfs [rw,noatime]...
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
ugen0.2: <vendor 0x8087> at usbus0
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen2.2: <vendor 0x8087> at usbus2
uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus2
uhub3: 6 ports with 6 removable, self powered
uhub4: 8 ports with 8 removable, self powered
GEOM_ELI: Device mirror/swap.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware

>How-To-Repeat:
I tried to analyze the size of a few snapshot size with zfs diff
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted:


More information about the freebsd-bugs mailing list