ZFS: zpool import hang on "zio->io_cv)" (with DDB output). Help
needed!
Johan Ström
johan at stromnet.se
Tue Mar 9 16:22:27 UTC 2010
Some followup, pjd tried to help me on IRC, without getting much
further on the actual problem.
Some more information:
Output from DDB PS: http://www.stromnet.se/~johan/back-1-ddb-ps.txt
Output from alltrace: http://www.stromnet.se/~johan/back-1-ddb-alltrace.txt
db> show alllocks
Process 2738 (sshd) thread 0xffffff0050418000 (100178)
Process 12 (intr) thread 0xffffff0002407ab0 (100018)
db> show lockedvnods
Locked vnodes
db >
Also tried accessing the disks (to make sure interrupts etc where
alive) ,using dd if=/dev/adXX of=/dev/null count=1 on every disk on
the system, works fine.
Rebooted the system and tried with zfs debugging enabled too, and
doing zpool import tank instead of just zpool import. no difference.
Hanging there now.
Johan
On Mar 9, 2010, at 15:26 , Johan Ström wrote:
> Hi List!
>
> I'm in the tedious process of upgrading my pool on my FreeBSD 8.0
> box (with a pool from 7.x). Yesterday I pulled a few disks from my
> mirrored pool (one pool with multiple mirrors), in order to be able
> to (free ports) plug in new ones and build my brand new pool.
> After removing disks, the pool got into state DEGRADED ofcourse,
> since some of the disks where gone, but data was still there. I did
> zpool detach on the removed disks, and the pool was ONLINE again,
> all fine!
> Then I rebooted to make another disk available through my old LSI
> MegaRaid card (didnt take time to figure out cryptic syntax of the
> megarc CLI util). On boot, I was met by something similar to this:
>
> pool: tank
> state: UNAVAIL
> status: One or more devices could not be used because the label is
> missing
> or invalid. There are insufficient replicas for the pool to
> continue
> functioning.
> action: Destroy and re-create the pool from a backup source.
> see: http://www.sun.com/msg/ZFS-8000-5E
> scrub: none requested
> config:
>
> NAME STATE READ WRITE CKSUM
> tank UNAVAIL 0 0 0 insufficient replicas
> ad10s1d ONLINE 0 0 0
> mirror DEGRADED 0 0 0
> ad12 FAULTED 0 0 0 corrupted data
> ad16 UNAVAIL 0 0 0 corrupted data
> mirror DEGRADED 0 0 0
> ad20 FAULTED 0 0 0 corrupted data
> ad18 UNAVAIL 0 0 0 corrupted data
> mirror UNAVAIL 0 0 0 insufficient replicas
> ad6 UNAVAIL 0 0 0 corrupted data
> ad4 FAULTED 0 0 0 corrupted data
> mirror ONLINE 0 0 0
> amrd0 ONLINE 0 0 0
> amrd1 ONLINE 0 0 0
>
> The reason the disks was still there was due to me haveing an old
> zfs.cache file in the boot (it boots from anothre drive, but thats
> another story..)
>
> In this case, a export/import did the trick, after reimporting, the
> pool was back online. All fine. I created a new pool and did some
> bonnie++ testing on it, and suddenly the box paniced or something (I
> didn't have dumpon enabled :/ and didn't see the screen until it
> rebooted)..
>
> Now my problems come. The box came up again, with the above output.
> I tried zpool export again, fine. Zpool import however, hang. Waited
> an hour, nothing. After rebuilding the kernel with DDB/witness and
> did the import again, i've managed to get this output (im not really
> sure what is usable here):
>
> back-1 # zpool import
> load: 0.29 cmd: zpool 3193 [zio->io_cv)] 2.54r 0.00u 0.01s 0% 2236k
>
>
>
> In DDB:
> > tr 3193
>
> Tracing pid 3193 tid 100122 td 0xffffff00035ca390
> sched_switch() at sched_switch+0xde
> mi_switch() at mi_switch+0x170
> sleepq_wait() at sleepq_wait+0x44
> _cv_wait() at _cv_wait+0x13c
> zio_wait() at zio_wait+0x61
> arc_read_nolock() at arc_read_nolock+0x345
> dmu_objset_open_impl() at dmu_objset_open_impl+0xd0
> dsl_pool_open() at dsl_pool_open+0x5a
> spa_load() at spa_load+0x31b
> spa_tryimport() at spa_tryimport+0xa9
> zfs_ioc_pool_tryimport() at zfs_ioc_pool_tryimport+0x3f
> zfsdev_ioctl() at zfsdev_ioctl+0x8d
> devfs_ioctl_f() at devfs_ioctl_f+0x76
> kern_ioctl() at kern_ioctl+0xf6
> ioctl() at ioctl+0xfd
> syscall() at syscall+0x19e
> Xfast_syscall() at Xfast_syscall+0xe1
> --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x8010eb86c, rsp =
> 0x7fffffff8e28, rbp = 0x801323300 ---
>
>
> db> show thread 100122
>
> Thread 100122 at 0xffffff00035ca390:
> proc (pid 3193): 0xffffff000383c460
> name: zpool
> stack: 0xffffff805740a000-0xffffff805740dfff
> flags: 0x44 pflags: 0x10000
> state: INHIBITED: {SLEEPING}
> wmesg: zio->io_cv) wchan: 0xffffff00506e5858
> priority: 131
> container lock: sleepq chain (0xffffffff80c61e68)
> db>
>
> db> show sleepchain 10012
>
> db> show sleepchain 3193
>
> thread 100122 (pid 3193, zpool) sleeping on 0xffffff00506e5858 "zio-
> >io_cv)"
> db> show lock 0xffffff00506e5858
>
> class: spin mutex
> name: zio->io_cv)
> flags: {SPIN}
> state: {OWNED}
>
>
>
>
> The box is currenlty in this state, so if you reply to me now I can
> continue to do debugging according to instructions.
> Since I cannot get anywhere on my own with this, all and any help is
> appreciated since I really need this pool to get online..
>
> dmesg is posted below (including some LOR?)
>
> Thanks!
> Johan
>
>
>
> DMESG:
>
> Copyright (c) 1992-2009 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
> 1994
> The Regents of the University of California. All rights
> reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 8.0-RELEASE-p2 #10: Tue Mar 9 12:44:15 CET 2010
> johan at back-1.stromnet.se:/usr/obj/usr/src/sys/BACK1
> WARNING: WITNESS option enabled, expect reduced performance.
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz (2666.68-MHz K8-
> class CPU)
> Origin = "GenuineIntel" Id = 0x6fb Stepping = 11
>
> Features
> =
> 0xbfebfbff
> <
> FPU
> ,VME
> ,DE
> ,PSE
> ,TSC
> ,MSR
> ,PAE
> ,MCE
> ,CX8
> ,APIC
> ,SEP
> ,MTRR
> ,PGE
> ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>
> Features2
> =0xe3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
> AMD Features=0x20100800<SYSCALL,NX,LM>
> AMD Features2=0x1<LAHF>
> TSC: P-state invariant
> real memory = 2147483648 (2048 MB)
> avail memory = 2040631296 (1946 MB)
> ACPI APIC Table: <GBT GBTUACPI>
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> FreeBSD/SMP: 1 package(s) x 2 core(s)
> cpu0 (BSP): APIC ID: 0
> cpu1 (AP): APIC ID: 1
> ioapic0: Changing APIC ID to 2
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> kbd1 at kbdmux0
> cryptosoft0: <software crypto> on motherboard
> acpi0: <GBT GBTUACPI> on motherboard
> acpi0: [ITHREAD]
> acpi0: Power Button (fixed)
> acpi0: reservation of 0, a0000 (3) failed
> acpi0: reservation of 100000, 7f4e0000 (3) failed
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
> acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff
> on acpi0
> Timecounter "HPET" frequency 14318180 Hz quality 900
> acpi_button0: <Power Button> on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> vgapci0: <VGA-compatible display> port 0xe000-0xe007 mem
> 0xe6300000-0xe637ffff,0xd0000000-0xdfffffff,0xe6000000-0xe60fffff
> irq 16 at device 2.0 on pci0
> agp0: <Intel G33 SVGA controller> on vgapci0
> agp0: detected 7164k stolen memory
> agp0: aperture size is 256M
> uhci0: <Intel 82801I (ICH9) USB controller> port 0xe100-0xe11f irq
> 16 at device 26.0 on pci0
> uhci0: [ITHREAD]
> uhci0: LegSup = 0x2f00
> usbus0: <Intel 82801I (ICH9) USB controller> on uhci0
> uhci1: <Intel 82801I (ICH9) USB controller> port 0xe200-0xe21f irq
> 21 at device 26.1 on pci0
> uhci1: [ITHREAD]
> uhci1: LegSup = 0x2f00
> usbus1: <Intel 82801I (ICH9) USB controller> on uhci1
> uhci2: <Intel 82801I (ICH9) USB controller> port 0xe600-0xe61f irq
> 18 at device 26.2 on pci0
> uhci2: [ITHREAD]
> uhci2: LegSup = 0x2f00
> usbus2: <Intel 82801I (ICH9) USB controller> on uhci2
> ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem
> 0xe6384000-0xe63843ff irq 18 at device 26.7 on pci0
> ehci0: [ITHREAD]
> usbus3: EHCI version 1.0
> usbus3: <Intel 82801I (ICH9) USB 2.0 controller> on ehci0
> hdac0: <Intel 82801I High Definition Audio Controller> mem
> 0xe6380000-0xe6383fff irq 22 at device 27.0 on pci0
> hdac0: HDA Driver Revision: 20090624_0136
> hdac0: [ITHREAD]
> pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> pcib2: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
> pci2: <ACPI PCI bus> on pcib2
> em0: <Intel(R) PRO/1000 Network Connection 6.9.14> port
> 0xa000-0xa01f mem 0xe1020000-0xe103ffff,0xe1000000-0xe101ffff irq 18
> at device 0.0 on pci2
> em0: Using MSI interrupt
> em0: [FILTER]
> em0: Ethernet address: 00:1b:21:05:00:b4
> pcib3: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
> pci3: <ACPI PCI bus> on pcib3
> atapci0: <JMicron JMB363 SATA300 controller> port
> 0xb000
> -0xb007,0xb100-0xb103,0xb200-0xb207,0xb300-0xb303,0xb400-0xb40f mem
> 0xe6100000-0xe6101fff irq 19 at device 0.0 on pci3
> atapci0: [ITHREAD]
> atapci0: AHCI called from vendor specific driver
> atapci0: AHCI v1.00 controller with 2 3Gbps ports, PM supported
> ata2: <ATA channel 0> on atapci0
> ata2: [ITHREAD]
> ata3: <ATA channel 1> on atapci0
> ata3: [ITHREAD]
> ata4: <ATA channel 2> on atapci0
> ata4: [ITHREAD]
> pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
> pci4: <ACPI PCI bus> on pcib4
> re0: <RealTek 8168/8168B/8168C/8168CP/8168D/8168DP/8111B/8111C/
> 8111CP/8111DP PCIe Gigabit Ethernet> port 0xc000-0xc0ff mem
> 0xe3000000-0xe3000fff irq 16 at device 0.0 on pci4
> re0: Using 1 MSI messages
> re0: Chip rev. 0x38000000
> re0: MAC rev. 0x00000000
> miibus0: <MII bus> on re0
> rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
> rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
> 1000baseT-FDX, auto
> re0: Ethernet address: 00:1a:4d:5a:97:87
> re0: [FILTER]
> uhci3: <Intel 82801I (ICH9) USB controller> port 0xe300-0xe31f irq
> 23 at device 29.0 on pci0
> uhci3: [ITHREAD]
> uhci3: LegSup = 0x2f00
> usbus4: <Intel 82801I (ICH9) USB controller> on uhci3
> uhci4: <Intel 82801I (ICH9) USB controller> port 0xe400-0xe41f irq
> 19 at device 29.1 on pci0
> uhci4: [ITHREAD]
> uhci4: LegSup = 0x2f00
> usbus5: <Intel 82801I (ICH9) USB controller> on uhci4
> uhci5: <Intel 82801I (ICH9) USB controller> port 0xe500-0xe51f irq
> 18 at device 29.2 on pci0
> uhci5: [ITHREAD]
> uhci5: LegSup = 0x2f00
> usbus6: <Intel 82801I (ICH9) USB controller> on uhci5
> ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem
> 0xe6385000-0xe63853ff irq 23 at device 29.7 on pci0
> ehci1: [ITHREAD]
> usbus7: EHCI version 1.0
> usbus7: <Intel 82801I (ICH9) USB 2.0 controller> on ehci1
> pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> pci5: <ACPI PCI bus> on pcib5
> amr0: <LSILogic MegaRAID 1.53> mem 0xe6200000-0xe620ffff irq 20 at
> device 0.0 on pci5
> amr0: Using 64-bit DMA
> amr0: [ITHREAD]
> amr0: delete logical drives supported by controller
> amr0: <LSILogic MegaRAID SATA 150-4D> Firmware 713S, BIOS G121, 64MB
> RAM
> xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xd000-0xd07f mem
> 0xe5004000-0xe500407f irq 19 at device 1.0 on pci5
> miibus1: <MII bus> on xl0
> xlphy0: <3c905C 10/100 internal PHY> PHY 24 on miibus1
> xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> xl0: Ethernet address: 00:04:76:ef:c6:36
> xl0: [ITHREAD]
> skc0: <SysKonnect Gigabit Ethernet (V1.0)> port 0xd100-0xd1ff mem
> 0xe5000000-0xe5003fff irq 18 at device 2.0 on pci5
> pci0:5:2:0: invalid VPD data, remain 0xfc
> skc0: SysKonnect SK-NET Gigabit Ethernet Adapter SK-9843 SX rev. (0x0)
> sk0: <XaQti Corp. XMAC II> on skc0
> sk0: Ethernet address: 00:00:5a:98:43:68
> miibus2: <MII bus> on sk0
> xmphy0: <XaQti Corp. XMAC II gigabit interface> PHY 0 on miibus2
> xmphy0: 1000baseSX, 1000baseSX-FDX, auto
> skc0: [ITHREAD]
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> atapci1: <Intel ICH9 SATA300 controller> port
> 0xe700
> -0xe707,0xe800-0xe803,0xe900-0xe907,0xea00-0xea03,0xeb00-0xeb1f mem
> 0xe6386000-0xe63867ff irq 19 at device 31.2 on pci0
> atapci1: [ITHREAD]
> atapci1: AHCI called from vendor specific driver
> atapci1: AHCI v1.20 controller with 6 3Gbps ports, PM supported
> ata5: <ATA channel 0> on atapci1
> ata5: [ITHREAD]
> ata6: <ATA channel 1> on atapci1
> ata6: [ITHREAD]
> ata7: <ATA channel 2> on atapci1
> ata7: [ITHREAD]
> ata8: <ATA channel 3> on atapci1
> ata8: [ITHREAD]
> ata9: <ATA channel 4> on atapci1
> ata9: [ITHREAD]
> ata10: <ATA channel 5> on atapci1
> ata10: [ITHREAD]
> pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
> atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2
> on acpi0
> fdc0: [FILTER]
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on
> acpi0
> uart0: [FILTER]
> ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
> ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
> ppc0: [ITHREAD]
> ppbus0: <Parallel port bus> on ppc0
> plip0: <PLIP network interface> on ppbus0
> plip0: [ITHREAD]
> lpt0: <Printer> on ppbus0
> lpt0: [ITHREAD]
> lpt0: Interrupt-driven port
> ppi0: <Parallel I/O> on ppbus0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> atkbd0: [ITHREAD]
> cpu0: <ACPI CPU> on acpi0
> est0: <Enhanced SpeedStep Frequency Control> on cpu0
> est: CPU supports Enhanced Speedstep, but is not recognized.
> est: cpu_vendor GenuineIntel, msr 82a082a0600082a
> device_attach: est0 attach returned 6
> p4tcc0: <CPU Frequency Thermal Control> on cpu0
> cpu1: <ACPI CPU> on acpi0
> est1: <Enhanced SpeedStep Frequency Control> on cpu1
> est: CPU supports Enhanced Speedstep, but is not recognized.
> est: cpu_vendor GenuineIntel, msr 82a082a0600082a
> device_attach: est1 attach returned 6
> p4tcc1: <CPU Frequency Thermal Control> on cpu1
> orm0: <ISA Option ROMs> at iomem 0xcc000-0xcc7ff,0xcd000-0xcefff on
> isa0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
> isa0
> ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM
> is present;
> to enable, add "vfs.zfs.prefetch_disable=0" to /boot/
> loader.conf.
> ZFS filesystem version 13
> ZFS storage pool version 13
> Timecounters tick every 1.000 msec
> IPsec: Initialized Security Association Processing.
> usbus0: 12Mbps Full Speed USB v1.0
> usbus1: 12Mbps Full Speed USB v1.0
> usbus2: 12Mbps Full Speed USB v1.0
> usbus3: 480Mbps High Speed USB v2.0
> usbus4: 12Mbps Full Speed USB v1.0
> usbus5: 12Mbps Full Speed USB v1.0
> usbus6: 12Mbps Full Speed USB v1.0
> usbus7: 480Mbps High Speed USB v2.0
> ad4: 476938MB <Seagate ST3500630NS 3.AEG> at ata2-master SATA300
> ugen0.1: <Intel> at usbus0
> uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
> usbus0
> ugen1.1: <Intel> at usbus1
> uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
> usbus1
> ugen2.1: <Intel> at usbus2
> uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
> usbus2
> ugen3.1: <Intel> at usbus3
> uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on
> usbus3
> ugen4.1: <Intel> at usbus4
> uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
> usbus4
> ugen5.1: <Intel> at usbus5
> uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
> usbus5
> ugen6.1: <Intel> at usbus6
> uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
> usbus6
> ugen7.1: <Intel> at usbus7
> uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on
> usbus7
> ad6: 1907729MB <WDC WD20EARS-00S8B1 80.00A80> at ata3-master SATA300
> uhub0: 2 ports with 2 removable, self powered
> uhub1: 2 ports with 2 removable, self powered
> uhub2: 2 ports with 2 removable, self powered
> uhub4: 2 ports with 2 removable, self powered
> uhub5: 2 ports with 2 removable, self powered
> uhub6: 2 ports with 2 removable, self powered
> ad10: 715404MB <WDC WD7500AAKS-00RBA0 30.04G30> at ata5-master SATA300
> ad12: 305245MB <Seagate ST3320620AS 3.AAE> at ata6-master SATA150
> ad14: 715404MB <WDC WD7500AAKS-00RBA0 30.04G30> at ata7-master SATA300
> ad16: 1907729MB <WDC WD20EARS-00S8B1 80.00A80> at ata8-master SATA300
> GEOM_MIRROR: Device mirror/gm1a launched (1/1).
> GEOM_MIRROR: Device mirror/gm1b launched (1/1).
> ad18: 1907729MB <WDC WD20EARS-00S8B1 80.00A80> at ata9-master SATA300
> GEOM_MIRROR: Device mirror/swap launched (1/1).
> ad20: 286187MB <Maxtor 7L300S0 BANC1G10> at ata10-master SATA150
> hdac0: HDA Codec #2: Realtek ALC885
> pcm0: <HDA Realtek ALC885 PCM #0 Analog> at cad 2 nid 1 on hdac0
> pcm1: <HDA Realtek ALC885 PCM #1 Analog> at cad 2 nid 1 on hdac0
> pcm2: <HDA Realtek ALC885 PCM #2 Analog> at cad 2 nid 1 on hdac0
> pcm3: <HDA Realtek ALC885 PCM #3 Analog> at cad 2 nid 1 on hdac0
> pcm4: <HDA Realtek ALC885 PCM #4 Digital> at cad 2 nid 1 on hdac0
> pcm5: <HDA Realtek ALC885 PCM #5 Analog> at cad 2 nid 1 on hdac0
> amr0: delete logical drives supported by controller
> amrd0: <LSILogic MegaRAID logical drive> on amr0
> amrd0: 476935MB (976762880 sectors) RAID 0 (optimal)
> amrd1: <LSILogic MegaRAID logical drive> on amr0
> amrd1: 476935MB (976762880 sectors) RAID 0 (optimal)
> amrd2: <LSILogic MegaRAID logical drive> on amr0
> amrd2: 1907724MB (3907018752 sectors) RAID 0 (optimal)
> SMP: AP CPU #1 Launched!
> WARNING: WITNESS option enabled, expect reduced performance.
> Root mount waiting for: usbus7 usbus3
> uhub3: 6 ports with 6 removable, self powered
> uhub7: 6 ports with 6 removable, self powered
> Trying to mount root from zfs:zroot
> ugen0.2: <Telldus> at usbus0
> ugen1.2: <American Power Conversion> at usbus1
> uma_zalloc_arg: zone "256" with the following non-sleepable locks
> held:
> exclusive rw ifnet_rw (ifnet_rw) r = 0 (0xffffffff80e01f60) locked
> @ /usr/src/sys/net/if.c:402
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2c
> witness_warn() at witness_warn+0x2c2
> uma_zalloc_arg() at uma_zalloc_arg+0x29d
> malloc() at malloc+0x5d
> if_grow() at if_grow+0x2f
> if_alloc() at if_alloc+0x2b3
> gif_clone_create() at gif_clone_create+0x53
> ifc_simple_create() at ifc_simple_create+0x89
> if_clone_createif() at if_clone_createif+0x64
> ifioctl() at ifioctl+0x6b5
> kern_ioctl() at kern_ioctl+0xf6
> ioctl() at ioctl+0xfd
> syscall() at syscall+0x19e
> Xfast_syscall() at Xfast_syscall+0xe1
> --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800b8286c, rsp =
> 0x7fffffffe4a8, rbp = 0x7fffffffef6e ---
> lock order reversal:
> 1st 0xffffffff80c093e0 pf task mtx (pf task mtx) @ /usr/src/sys/
> contrib/pf/net/pf_ioctl.c:1393
> 2nd 0xffffffff80e01f60 ifnet_rw (ifnet_rw) @ /usr/src/sys/net/if.c:
> 2034
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2c
> witness_checkorder() at witness_checkorder+0x66f
> _rw_rlock() at _rw_rlock+0x29
> ifunit() at ifunit+0x22
> pfioctl() at pfioctl+0x262a
> devfs_ioctl_f() at devfs_ioctl_f+0x76
> kern_ioctl() at kern_ioctl+0xf6
> ioctl() at ioctl+0xfd
> syscall() at syscall+0x19e
> Xfast_syscall() at Xfast_syscall+0xe1
> --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x80099886c, rsp =
> 0x7fffffffdb68, rbp = 0x7fffffffdc20 ---
> lock order reversal:
> 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1054
> 2nd 0xffffff005010f448 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:
> 2083
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2c
> witness_checkorder() at witness_checkorder+0x66f
> __lockmgr_args() at __lockmgr_args+0x475
> vop_stdlock() at vop_stdlock+0x39
> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46
> _vn_lock() at _vn_lock+0x47
> vget() at vget+0x56
> devfs_allocv() at devfs_allocv+0x103
> devfs_root() at devfs_root+0x48
> vfs_donmount() at vfs_donmount+0xf43
> nmount() at nmount+0x63
> syscall() at syscall+0x19e
> Xfast_syscall() at Xfast_syscall+0xe1
> --- syscall (378, FreeBSD ELF64, nmount), rip = 0x8007b04dc, rsp =
> 0x7fffffffdd28, rbp = 0x800a04048 ---
> n
>
> r
> tun0: link state changed to UP
> lock order reversal:
> 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1200
> 2nd 0xffffff005010f270 syncer (syncer) @ /usr/src/sys/kern/
> vfs_subr.c:2188
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2c
> witness_checkorder() at witness_checkorder+0x66f
> __lockmgr_args() at __lockmgr_args+0x475
> vop_stdlock() at vop_stdlock+0x39
> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46
> _vn_lock() at _vn_lock+0x47
> vrele() at vrele+0xc3
> dounmount() at dounmount+0x269
> unmount() at unmount+0x27e
> syscall() at syscall+0x19e
> Xfast_syscall() at Xfast_syscall+0xe1
> --- syscall (22, FreeBSD ELF64, unmount), rip = 0x8006a09bc, rsp =
> 0x7fffffffde18, rbp = 0 ---
> KDB: enter: manual escape to debugger
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
More information about the freebsd-fs
mailing list