kern/127439: deadlock in pf
Christian Peron
csjp at freebsd.org
Wed Sep 17 16:50:05 UTC 2008
The following reply was made to PR kern/127439; it has been noted by GNATS.
From: Christian Peron <csjp at freebsd.org>
To: Geoffrey Mainland <mainland at apeiron.net>
Cc: FreeBSD-gnats-submit at freebsd.org
Subject: Re: kern/127439: deadlock in pf
Date: Wed, 17 Sep 2008 11:16:01 -0500
Can you provide a copy of your pf ruleset?
On Wed, Sep 17, 2008 at 08:33:23AM -0400, Geoffrey Mainland wrote:
>
> >Number: 127439
> >Category: kern
> >Synopsis: deadlock in pf
> >Confidential: no
> >Severity: critical
> >Priority: high
> >Responsible: freebsd-bugs
> >State: open
> >Quarter:
> >Keywords:
> >Date-Required:
> >Class: sw-bug
> >Submitter-Id: current-users
> >Arrival-Date: Wed Sep 17 12:50:01 UTC 2008
> >Closed-Date:
> >Last-Modified:
> >Originator: Geoffrey Mainland
> >Release: FreeBSD 7.1-PRERELEASE i386
> >Organization:
> >Environment:
> System: FreeBSD zeno.apeiron.net 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #7: Tue Sep 16 09:28:16 EDT 2008 toor at zeno.apeiron.net:/usr/obj/usr/src/sys/ZENO i386
>
>
> >Description:
>
> This happens reliably every night. I'm not sure what's running that triggers it.
>
> ifconfig:
>
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
> ether 00:0e:0c:5f:c1:f8
> inet6 fe80::20e:cff:fe5f:c1f8%em0 prefixlen 64 scopeid 0x1
> inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255
> inet 192.168.0.1 netmask 0xffffffff broadcast 192.168.0.1
> inet 192.168.0.2 netmask 0xffffffff broadcast 192.168.0.2
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=8<VLAN_MTU>
> ether 00:90:27:62:87:4d
> inet6 fe80::290:27ff:fe62:874d%fxp0 prefixlen 64 scopeid 0x2
> inet 68.164.219.98 netmask 0xfffffff8 broadcast 68.164.219.103
> inet 68.164.219.99 netmask 0xffffffff broadcast 68.164.219.99
> inet 68.164.219.100 netmask 0xffffffff broadcast 68.164.219.100
> inet 68.164.219.101 netmask 0xffffffff broadcast 68.164.219.101
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=2808<VLAN_MTU,WOL_UCAST,WOL_MAGIC>
> ether 00:15:f2:43:48:7b
> inet6 fe80::215:f2ff:fe43:487b%vr0 prefixlen 64 scopeid 0x3
> inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
> inet 192.168.1.2 netmask 0xffffffff broadcast 192.168.1.2
> media: Ethernet autoselect (none)
> status: no carrier
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
> inet 127.0.0.1 netmask 0xff000000
> pfsync0: flags=0<> metric 0 mtu 1460
> syncpeer: 224.0.0.240 maxupd: 128
> pflog0: flags=0<> metric 0 mtu 33204
> gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
> tunnel inet 68.164.219.98 --> 66.55.128.25
> inet6 fe80::20e:cff:fe5f:c1f8%gif0 prefixlen 64 scopeid 0x7
> inet6 2001:4830:1200:10b::2 --> 2001:4830:1200:10b::1 prefixlen 128
> tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
> inet6 fe80::20e:cff:fe5f:c1f8%tun0 prefixlen 64 scopeid 0x8
> inet 192.168.2.1 --> 192.168.2.2 netmask 0xffffffff
> Opened by PID 1454
>
> Kernel config:
>
> cpu I686_CPU
> ident ZENO
> options SCHED_ULE
> options SMP
> options PREEMPTION
> options DEVICE_POLLING
> options HZ=2000
> options _KPOSIX_PRIORITY_SCHEDULING
> options P1003_1B_MQUEUE
> options KDB
> options KDB_TRACE
> options DDB
> options WITNESS
> options INVARIANTS
> options INVARIANT_SUPPORT
> makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols
> options COMPAT_FREEBSD4
> options COMPAT_FREEBSD5
> options COMPAT_FREEBSD6
> options SYSVSHM
> options SYSVSEM
> options SYSVMSG
> options STACK
> options INET #Internet communications protocols
> options INET6 #IPv6 communications protocols
> options IPSEC #IP security (requires device crypto)
> options NETATALK #Appletalk communications protocols
> options NETSMB #SMB/CIFS requester
> options LIBMCHAIN
> options SCTP
> options NETGRAPH # netgraph(4) system
> device ether #Generic Ethernet
> device loop #Network loopback device
> device bpf #Berkeley packet filter
> device tap #Virtual Ethernet driver
> device tun #Tunnel driver (ppp(8), nos-tun(8))
> device gre #IP over IP tunneling
> device pf #PF OpenBSD packet-filter firewall
> device pflog #logging support interface for PF
> device pfsync #synchronization interface for PF
> device gif #IPv6 and IPv4 tunneling
> device faith #for IPv6 and IPv4 translation
> device stf #6to4 IPv6 over IPv4 encapsulation
> options FFS #Fast filesystem
> options NFSCLIENT #Network File System client
> options CD9660 #ISO 9660 filesystem
> options MSDOSFS #MS DOS File System (FAT, FAT32)
> options NFSSERVER #Network File System server
> options NFSLOCKD #Network Lock Manager
> options NTFS #NT File System
> options PROCFS #Process filesystem (requires PSEUDOFS)
> options PSEUDOFS #Pseudo-filesystem framework
> options SMBFS #SMB/CIFS filesystem
> options UDF #Universal Disk Format
> options NFS_ROOT #NFS usable as root device
> options SOFTUPDATES
> options UFS_ACL
> options UFS_DIRHASH
> device random
> device mem
> options AUDIT
> device scbus #base SCSI code
> device da #SCSI direct access devices (aka disks)
> device cd #SCSI CD-ROMs
> device pt #SCSI processor
> device pass #CAM passthrough driver
> device pty #Pseudo ttys
> device md #Memory/malloc disk
> options LIBICONV
> options KBD_INSTALL_CDEV # install a CDEV entry in /dev
> device splash # Splash screen and screen saver support
> device sc
> options SC_DISABLE_KDBKEY # disable `debug' key
> device ata
> device atadisk # ATA disk drives
> device ataraid # ATA RAID drives
> device atapicd # ATAPI CDROM drives
> device atapifd # ATAPI floppy drives
> device atapicam # emulate ATAPI devices as SCSI ditto via CAM
> options ATA_STATIC_ID
> device fdc
> device sound
> device ppc
> device ppbus
> device lpt
> device ppi
> device uhci
> device ehci
> device usb
> device crypto # core crypto support
> device cryptodev # /dev/crypto for access to h/w
> device apic # I/O apic
> device nvram # Access to rtc cmos via /dev/nvram
> device sio
> device eisa
> device pci
> options VESA
> device psm
> device atkbdc
> device atkbd
> device vga
> options COMPAT_LINUX
> options COMPAT_AOUT
> options LINPROCFS
> options LINSYSFS
>
>
>
>
>
> dmesg output (after crash):
>
> Copyright (c) 1992-2008 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 7.1-PRERELEASE #7: Tue Sep 16 09:28:16 EDT 2008
> toor at zeno.apeiron.net:/usr/obj/usr/src/sys/ZENO
> WARNING: WITNESS option enabled, expect reduced performance.
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: AMD Sempron(tm) Processor 3100+ (1800.09-MHz 686-class CPU)
> Origin = "AuthenticAMD" Id = 0x10fc0 Stepping = 0
> Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
> AMD Features=0xc2500800<SYSCALL,NX,MMX+,FFXSR,3DNow!+,3DNow!>
> AMD Features2=0x1<LAHF>
> real memory = 1073414144 (1023 MB)
> avail memory = 1040887808 (992 MB)
> WITNESS: spin lock cpuset not in order list
> WITNESS: spin lock intrcnt not in order list
> netsmb_dev: loaded
> cryptosoft0: <software crypto> on motherboard
> acpi0: <A M I OEMRSDT> on motherboard
> acpi0: [ITHREAD]
> acpi0: Power Button (fixed)
> acpi0: reservation of 0, a0000 (3) failed
> acpi0: reservation of 100000, 3fef0000 (3) failed
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> vgapci0: <VGA-compatible display> mem
> 0xfb000000-0xfbffffff,0xf0000000-0xf7ffffff irq 11 at device 0.0 on pci1
> em0: <Intel(R) PRO/1000 Network Connection 6.9.5> port 0xe800-0xe83f mem
> 0xfae00000-0xfae1ffff,0xfad00000-0xfad1ffff irq 11 at device 11.0 on pci0
> em0: [FILTER]
> em0: Ethernet address: 00:0e:0c:5f:c1:f8
> fxp0: <Intel 82559 Pro/100 Ethernet> port 0xe400-0xe43f mem
> 0xfab00000-0xfab00fff,0xfaa00000-0xfaafffff irq 10 at device 12.0 on pci0
> miibus0: <MII bus> on fxp0
> inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
> inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> fxp0: Ethernet address: 00:90:27:62:87:4d
> fxp0: [ITHREAD]
> atapci0: <VIA 6420 SATA150 controller> port
> 0xe000-0xe007,0xd800-0xd803,0xd400-0xd407,0xd000-0xd003,0xc800-0xc80f,0xc400-0xc4ff
> irq 10 at device 15.0 on pci0
> atapci0: [ITHREAD]
> ata2: <ATA channel 0> on atapci0
> ata2: [ITHREAD]
> ata3: <ATA channel 1> on atapci0
> ata3: [ITHREAD]
> atapci1: <VIA 8237 UDMA133 controller> port
> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 15.1 on pci0
> ata0: <ATA channel 0> on atapci1
> ata0: [ITHREAD]
> ata1: <ATA channel 1> on atapci1
> ata1: [ITHREAD]
> uhci0: <VIA 83C572 USB controller> port 0xb000-0xb01f irq 11 at device 16.0 on
> pci0
> uhci0: [GIANT-LOCKED]
> uhci0: [ITHREAD]
> usb0: <VIA 83C572 USB controller> on uhci0
> usb0: USB revision 1.0
> uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
> uhub0: 2 ports with 2 removable, self powered
> uhci1: <VIA 83C572 USB controller> port 0xb400-0xb41f irq 11 at device 16.1 on
> pci0
> uhci1: [GIANT-LOCKED]
> uhci1: [ITHREAD]
> usb1: <VIA 83C572 USB controller> on uhci1
> usb1: USB revision 1.0
> uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
> uhub1: 2 ports with 2 removable, self powered
> uhci2: <VIA 83C572 USB controller> port 0xb800-0xb81f irq 10 at device 16.2 on
> pci0
> uhci2: [GIANT-LOCKED]
> uhci2: [ITHREAD]
> usb2: <VIA 83C572 USB controller> on uhci2
> usb2: USB revision 1.0
> uhub2: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
> uhub2: 2 ports with 2 removable, self powered
> uhci3: <VIA 83C572 USB controller> port 0xc000-0xc01f irq 10 at device 16.3 on
> pci0
> uhci3: [GIANT-LOCKED]
> uhci3: [ITHREAD]
> usb3: <VIA 83C572 USB controller> on uhci3
> usb3: USB revision 1.0
> uhub3: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
> uhub3: 2 ports with 2 removable, self powered
> ehci0: <VIA VT6202 USB 2.0 controller> mem 0xfa700000-0xfa7000ff irq 5 at device
> 16.4 on pci0
> ehci0: [GIANT-LOCKED]
> ehci0: [ITHREAD]
> usb4: EHCI version 1.0
> usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
> usb4: <VIA VT6202 USB 2.0 controller> on ehci0
> usb4: USB revision 2.0
> uhub4: <VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
> uhub4: 8 ports with 8 removable, self powered
> isab0: <PCI-ISA bridge> at device 17.0 on pci0
> isa0: <ISA bus> on isab0
> pci0: <multimedia, audio> at device 17.5 (no driver attached)
> vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xa400-0xa4ff mem
> 0xfa600000-0xfa6000ff irq 11 at device 18.0 on pci0
> vr0: Quirks: 0x0
> vr0: Revision: 0x78
> miibus1: <MII bus> on vr0
> rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus1
> rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> vr0: Ethernet address: 00:15:f2:43:48:7b
> vr0: [ITHREAD]
> cpu0: <ACPI CPU> on acpi0
> acpi_button0: <Power Button> on acpi0
> acpi_button1: <Sleep Button> on acpi0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> atkbd0: [ITHREAD]
> fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
> acpi0
> fdc0: [FILTER]
> fd0: <1440-KB 3.5" drive> on fdc0 drive 0
> sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> sio0: type 16550A
> sio0: [FILTER]
> orm0: <ISA Option ROMs> at iomem 0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xd3fff
> pnpid ORM0000 on isa0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
> ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
> ppc0: FIFO with 16/16/9 bytes threshold
> ppbus0: <Parallel port bus> on ppc0
> ppbus0: [ITHREAD]
> lpt0: <Printer> on ppbus0
> lpt0: Interrupt-driven port
> ppi0: <Parallel I/O> on ppbus0
> ppc0: [GIANT-LOCKED]
> ppc0: [ITHREAD]
> sio1: configured irq 3 not in bitmap of probed irqs 0
> sio1: port may not be enabled
> Timecounter "TSC" frequency 1800086355 Hz quality 800
> Timecounters tick every 1.000 msec
> IPsec: Initialized Security Association Processing.
> ad0: 194481MB <Maxtor 6B200P0 BAH41BM0> at ata0-master UDMA133
> acd0: DVDR <NEC DVD RW ND-3550A/1.05> at ata1-master UDMA33
> ad4: 239372MB <Maxtor 7L250S0 BANC1G10> at ata2-master SATA150
> cd0 at ata1 bus 0 target 0 lun 0
> cd0: <_NEC DVD_RW ND-3550A 1.05> Removable CD-ROM SCSI-0 device
> cd0: 33.000MB/s transfers
> cd0: Attempt to query device size failed: NOT READY, Medium not present
> WARNING: WITNESS option enabled, expect reduced performance.
> Trying to mount root from ufs:/dev/ad4s1a
> WARNING: / was not properly dismounted
> lock order reversal:
> 1st 0xc0907fcc pf task mtx (pf task mtx) @
> /usr/src/sys/contrib/pf/net/pf_ioctl.c:1394
> 2nd 0xc0973488 ifnet (ifnet) @ /usr/src/sys/net/if.c:1558
> KDB: stack backtrace:
> db_trace_self_wrapper(c088cf61,e658ba3c,c05eb7b6,c088f4ad,c0973488,...) at
> db_trace_self_wrapper+0x26
> kdb_backtrace(c088f4ad,c0973488,c0896cfd,c0896cfd,c0896b56,...) at
> kdb_backtrace+0x29
> witness_checkorder(c0973488,9,c0896b56,616,0,...) at witness_checkorder+0x6d6
> _mtx_lock_flags(c0973488,0,c0896b56,616,c3f37a70,...) at _mtx_lock_flags+0xbc
> ifunit(c3f37a70,0,c08711f2,572,c05e958e,...) at ifunit+0x2f
> pfioctl(c3d2d800,c0104414,c3f37a70,3,c3f48690,...) at pfioctl+0x23b5
> devfs_ioctl_f(c3f49c2c,c0104414,c3f37a70,c3b2c000,c3f48690,...) at
> devfs_ioctl_f+0xe5
> kern_ioctl(c3f48690,3,c0104414,c3f37a70,1000000,...) at kern_ioctl+0x243
> ioctl(c3f48690,e658bcfc,c,c08bade8,c08d3630,...) at ioctl+0x134
> syscall(e658bd38) at syscall+0x274
> Xint0x80_syscall() at Xint0x80_syscall+0x20
> --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x281aac4b, esp = 0xbfbfde5c, ebp
> = 0xbfbfde88 ---
> lock order reversal:
> 1st 0xc097830c tcp (tcp) @ /usr/src/sys/netinet/tcp_input.c:400
> 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @
> /usr/src/sys/net/pfil.c:73
> KDB: stack backtrace:
> db_trace_self_wrapper(c088cf61,e42579ac,c05eb7b6,c088f4ad,c09775d8,...) at
> db_trace_self_wrapper+0x26
> kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at
> kdb_backtrace+0x29
> witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at
> witness_checkorder+0x6d6
> _rw_rlock(c09775d8,c0897d93,49,e4257a6c,0,...) at _rw_rlock+0x8e
> pfil_run_hooks(c09775c0,e4257a8c,c3c31c00,2,0,...) at pfil_run_hooks+0x35
> ip_output(c3c46100,0,e4257a50,0,0,0,c08e7c90,0,0,0,c067c807,c08e7c94,c08e7c9c,c8)
> at ip_output+0x90f
> tcp_respond(0,c3c87020,c3c87034,c3c46100,2da9088c,...) at tcp_respond+0x3e7
> tcp_dropwithreset(1,3,c089c953,353,1900,...) at tcp_dropwithreset+0x152
> tcp_input(c3c46100,14,c3c31c00,1,0,...) at tcp_input+0xe45
> ip_input(c3c46100,c3c46100,800,c3c31c00,800,...) at ip_input+0x686
> netisr_dispatch(2,c3c46100,10,3,0,...) at netisr_dispatch+0x72
> ether_demux(c3c31c00,c3c46100,3,0,3,...) at ether_demux+0x2e5
> ether_input(c3c31c00,c3c46100,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f
> fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4
> fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0
> ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5
> fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8
> fork_trampoline() at fork_trampoline+0x8
> --- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 ---
> lock order reversal:
> 1st 0xc4013d44 udpinp (udpinp) @ /usr/src/sys/netinet/udp_usrreq.c:878
> 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @
> /usr/src/sys/net/pfil.c:73
> KDB: stack backtrace:
> db_trace_self_wrapper(c088cf61,e658ba14,c05eb7b6,c088f4ad,c09775d8,...) at
> db_trace_self_wrapper+0x26
> kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at
> kdb_backtrace+0x29
> witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at
> witness_checkorder+0x6d6
> _rw_rlock(c09775d8,c0897d93,49,e658bad4,c4013ca8,...) at _rw_rlock+0x8e
> pfil_run_hooks(c09775c0,e658baf4,c3d44000,2,c4013ca8,...) at pfil_run_hooks+0x35
> ip_output(c3ef6100,0,e658bab8,0,0,...) at ip_output+0x90f
> udp_send(c42454e0,0,c3ef6100,0,0,...) at udp_send+0x8cd
> sosend_dgram(c42454e0,0,e658bbec,c3ef6100,0,...) at sosend_dgram+0x351
> sosend(c42454e0,0,e658bbec,0,0,...) at sosend+0x54
> kern_sendit(c3f48690,4,e658bc68,0,0,...) at kern_sendit+0xdb
> sendit(0,8143023,0,0,0,...) at sendit+0xb1
> sendto(c3f48690,e658bcfc,18,c08a5d78,c08d3d98,...) at sendto+0x48
> syscall(e658bd38) at syscall+0x274
> Xint0x80_syscall() at Xint0x80_syscall+0x20
> --- syscall (133, FreeBSD ELF32, sendto), eip = 0x2816bc83, esp = 0xbfbfd73c,
> ebp = 0xbfbfd768 ---
> lock order reversal:
> 1st 0xc423f150 tcpinp (tcpinp) @ /usr/src/sys/netinet/tcp_usrreq.c:472
> 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @
> /usr/src/sys/net/pfil.c:73
> KDB: stack backtrace:
> db_trace_self_wrapper(c088cf61,e65a3a30,c05eb7b6,c088f4ad,c09775d8,...) at
> db_trace_self_wrapper+0x26
> kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at
> kdb_backtrace+0x29
> witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at
> witness_checkorder+0x6d6
> _rw_rlock(c09775d8,c0897d93,49,e65a3af0,c423f0b4,...) at _rw_rlock+0x8e
> pfil_run_hooks(c09775c0,e65a3b10,c3d44000,2,c423f0b4,...) at pfil_run_hooks+0x35
> ip_output(c3c94e00,0,e65a3ad4,0,0,...) at ip_output+0x90f
> tcp_output(c42421d0,c3d2bc50,1d8,c423f150,c4259000,...) at tcp_output+0x140c
> tcp_usr_connect(c4259000,c3d2bc50,c3d2f8c0,25,e65a3c64,...) at
> tcp_usr_connect+0x11c
> soconnect(c4259000,c3d2bc50,c3d2f8c0,10,16,...) at soconnect+0x52
> kern_connect(c3d2f8c0,9,c3d2bc50,c3d2bc50,0,...) at kern_connect+0x59
> connect(c3d2f8c0,e65a3cfc,c,c088ff65,c08d3a50,...) at connect+0x46
> syscall(e65a3d38) at syscall+0x274
> Xint0x80_syscall() at Xint0x80_syscall+0x20
> --- syscall (98, FreeBSD ELF32, connect), eip = 0x28161e9b, esp = 0xbfbfe71c,
> ebp = 0xbfbfe868 ---
> lock order reversal:
> 1st 0xc3eda524 tcp_sc_head (tcp_sc_head) @
> /usr/src/sys/netinet/tcp_syncache.c:494
> 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @
> /usr/src/sys/net/pfil.c:73
> KDB: stack backtrace:
> db_trace_self_wrapper(c088cf61,e4257854,c05eb7b6,c088f4ad,c09775d8,...) at
> db_trace_self_wrapper+0x26
> kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at
> kdb_backtrace+0x29
> witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at
> witness_checkorder+0x6d6
> _rw_rlock(c09775d8,c0897d93,49,e4257914,0,...) at _rw_rlock+0x8e
> pfil_run_hooks(c09775c0,e4257934,c3c31c00,2,0,...) at pfil_run_hooks+0x35
> ip_output(c3ef7a00,0,e42578f8,0,0,...) at ip_output+0x90f
> syncache_respond(c426ad70,c40c0834,0,0,c40c0834,...) at syncache_respond+0x3a2
> _syncache_add(c42400b4,e4257ba8,c40b3700,0,0,...) at _syncache_add+0x2b0
> syncache_add(e4257b68,e4257b90,c40c0834,c42400b4,e4257ba8,...) at
> syncache_add+0x38
> tcp_input(c40b3700,14,c3c31c00,1,0,...) at tcp_input+0xd6b
> ip_input(c40b3700,c40b3700,800,c3c31c00,800,...) at ip_input+0x686
> netisr_dispatch(2,c40b3700,10,3,0,...) at netisr_dispatch+0x72
> ether_demux(c3c31c00,c40b3700,3,0,3,...) at ether_demux+0x2e5
> ether_input(c3c31c00,c40b3700,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f
> fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4
> fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0
> ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5
> fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8
> fork_trampoline() at fork_trampoline+0x8
> --- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 ---
> lock order reversal:
> 1st 0xc09786cc udp (udp) @ /usr/src/sys/netinet/udp_usrreq.c:395
> 2nd 0xc09775d8 PFil hook read/write mutex (PFil hook read/write mutex) @
> /usr/src/sys/net/pfil.c:73
> KDB: stack backtrace:
> db_trace_self_wrapper(c088cf61,e42579b8,c05eb7b6,c088f4ad,c09775d8,...) at
> db_trace_self_wrapper+0x26
> kdb_backtrace(c088f4ad,c09775d8,c0897dab,c0897dab,c0897d93,...) at
> kdb_backtrace+0x29
> witness_checkorder(c09775d8,1,c0897d93,49,c08a1d09,...) at
> witness_checkorder+0x6d6
> _rw_rlock(c09775d8,c0897d93,49,e4257a78,0,...) at _rw_rlock+0x8e
> pfil_run_hooks(c09775c0,e4257a98,c3c31c00,2,0,...) at pfil_run_hooks+0x35
> ip_output(c3efae00,0,e4257a5c,0,0,...) at ip_output+0x90f
> icmp_reflect(c40c6020,c3efaec8,14,c3efaf00,c40c6020,...) at icmp_reflect+0x3df
> icmp_error(c40b4d00,3,3,0,0,...) at icmp_error+0x3bd
> udp_input(c40b4d00,14,c3c31c00,1,0,...) at udp_input+0x5ea
> ip_input(c40b4d00,c40b4d00,800,c3c31c00,800,...) at ip_input+0x686
> netisr_dispatch(2,c40b4d00,10,3,0,...) at netisr_dispatch+0x72
> ether_demux(c3c31c00,c40b4d00,3,0,3,...) at ether_demux+0x2e5
> ether_input(c3c31c00,c40b4d00,c0aa0a74,6a9,ffffffff,...) at ether_input+0x37f
> fxp_intr_body(ffffffff,0,c0aa0a74,5db,c3c33014,...) at fxp_intr_body+0x1c4
> fxp_intr(c3c33000,0,c08866ae,4b6,c3b3c268,...) at fxp_intr+0xa0
> ithread_loop(c3c1fa50,e4257d38,c0886453,31c,c3bef2b8,...) at ithread_loop+0x1c5
> fork_exit(c0590660,c3c1fa50,e4257d38) at fork_exit+0xb8
> fork_trampoline() at fork_trampoline+0x8
> --- trap 0, eip = 0, esp = 0xe4257d70, ebp = 0 ---
>
>
>
>
>
> kernel backtrace:
>
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "i386-marcel-freebsd"...
>
> Unread portion of the kernel message buffer:
> panic: _rw_rlock (tcp): wlock already held @
> /usr/src/sys/contrib/pf/net/pf.c:3016
> cpuid = 0
> KDB: stack backtrace:
> db_trace_self_wrapper(c088cf61,e6846220,c05ae7df,c08b659d,0,...) at
> db_trace_self_wrapper+0x26
> kdb_backtrace(c08b659d,0,c0889c7e,e684622c,0,...) at kdb_backtrace+0x29
> panic(c0889c7e,c085a754,c088f55e,c087092d,bc8,...) at panic+0x10f
> _rw_rlock(c097830c,c087092d,bc8,c08d9624,c087092d,...) at _rw_rlock+0x73
> pf_socket_lookup(2,e68463dc,0,cc4,3,...) at pf_socket_lookup+0x208
> pf_test_tcp(e6846444,e6846440,2,c3efee00,c3c8e900,...) at pf_test_tcp+0x142
> pf_test6(2,c3d44000,e68464a0,0,0,...) at pf_test6+0x8a0
> pf_check6_out(0,e68464a0,c3d44000,2,0,...) at pf_check6_out+0x47
> pfil_run_hooks(c097ad00,e6846638,c3d44000,2,0,...) at pfil_run_hooks+0x88
> ip6_output(c3c8e900,0,e6846618,0,0,...) at ip6_output+0x122e
> pf_send_tcp(c4fcfe00,c41259b4,1c,c4fcfe5c,c4fcfe4c,...) at pf_send_tcp+0x6dd
> pf_test_tcp(e68468e8,e68468e4,2,c3f20900,c4fcfe00,...) at pf_test_tcp+0xcef
> pf_test6(2,c3f06400,e6846944,0,c446b7bc,...) at pf_test6+0x8a0
> pf_check6_out(0,e6846944,c3f06400,2,c446b7bc,...) at pf_check6_out+0x47
> pfil_run_hooks(c097ad00,e6846adc,c3f06400,2,c446b7bc,...) at pfil_run_hooks+0x88
> ip6_output(c4fcfe00,0,e6846abc,0,0,...) at ip6_output+0x122e
> tcp_output(c45553a0,c447e7c0,201,c446b858,c45553a0,...) at tcp_output+0x137e
> tcp6_usr_connect(c50cd340,c447e7c0,c4eed690,25,e6846c64,...) at
> tcp6_usr_connect+0x171
> soconnect(c50cd340,c447e7c0,c4eed690,1c,16,...) at soconnect+0x52
> kern_connect(c4eed690,3,c447e7c0,c447e7c0,0,...) at kern_connect+0x59
> connect(c4eed690,e6846cfc,c,c08a288e,c08d3a50,...) at connect+0x46
> syscall(e6846d38) at syscall+0x274
> Xint0x80_syscall() at Xint0x80_syscall+0x20
> --- syscall (98, FreeBSD ELF32, connect), eip = 0x282e6e9b, esp = 0xbfbfe7ec,
> ebp = 0xbfbfe848 ---
> KDB: enter: panic
> shared rw PFil hook read/write mutex r = 1 (0xc097ad18) locked @
> /usr/src/sys/net/pfil.c:73
> exclusive rw tcpinp r = 0 (0xc446b858) locked @
> /usr/src/sys/netinet/tcp_usrreq.c:513
> exclusive rw tcp r = 0 (0xc097830c) locked @
> /usr/src/sys/netinet/tcp_usrreq.c:510
> exclusive sx so_rcv_sx r = 0 (0xc452fbec) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc483cbec) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc4e89bec) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc4e8970c) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc483c22c) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc480d70c) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc4e8a08c) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc4e8a56c) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc41a456c) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc41c156c) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc41c18ac) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc41c1bec) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> exclusive sx so_rcv_sx r = 0 (0xc41f108c) locked @
> /usr/src/sys/kern/uipc_sockbuf.c:148
> shared rw udpinp r = 0 (0xc400f63c) locked @
> /usr/src/sys/netinet/udp_usrreq.c:878
> Uptime: 16h23m36s
> Physical memory: 1015 MB
> Dumping 166 MB: 151 135 119 103 87 71 55 39 23 7
>
> Reading symbols from /boot/kernel/if_em.ko...Reading symbols from
> /boot/kernel/if_em.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/if_em.ko
> Reading symbols from /boot/kernel/if_fxp.ko...Reading symbols from
> /boot/kernel/if_fxp.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/if_fxp.ko
> Reading symbols from /boot/kernel/miibus.ko...Reading symbols from
> /boot/kernel/miibus.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/miibus.ko
> Reading symbols from /boot/kernel/if_vr.ko...Reading symbols from
> /boot/kernel/if_vr.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/if_vr.ko
> Reading symbols from /boot/kernel/ulpt.ko...Reading symbols from
> /boot/kernel/ulpt.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/ulpt.ko
> Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from
> /boot/kernel/accf_http.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/accf_http.ko
> Reading symbols from /boot/kernel/acpi.ko...Reading symbols from
> /boot/kernel/acpi.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/acpi.ko
> #0 doadump () at pcpu.h:196
> 196 pcpu.h: No such file or directory.
> in pcpu.h
> (kgdb) bt
> #0 doadump () at pcpu.h:196
> #1 0xc05ae54c in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
> #2 0xc05ae816 in panic (fmt=Variable "fmt" is not available.
> ) at /usr/src/sys/kern/kern_shutdown.c:572
> #3 0xc05acf63 in _rw_rlock (rw=0xc097830c, file=0xc087092d
> "/usr/src/sys/contrib/pf/net/pf.c", line=3016)
> at /usr/src/sys/kern/kern_rwlock.c:253
> #4 0xc0473e58 in pf_socket_lookup (direction=2, pd=0xe68463dc, inp_arg=0x0) at
> /usr/src/sys/contrib/pf/net/pf.c:3016
> #5 0xc047dd62 in pf_test_tcp (rm=0xe6846444, sm=0xe6846440, direction=2,
> kif=0xc3efee00, m=0xc3c8e900, off=40,
> h=0xc3c8e944, pd=0xe68463dc, am=0xe6846448, rsm=0xe684643c, ifq=0x0,
> inp=0x0)
> at /usr/src/sys/contrib/pf/net/pf.c:3270
> #6 0xc04816c0 in pf_test6 (dir=2, ifp=0xc3d44000, m0=0xe68464a0, eh=0x0,
> inp=0x0)
> at /usr/src/sys/contrib/pf/net/pf.c:7368
> #7 0xc0484e37 in pf_check6_out (arg=0x0, m=0xe68464a0, ifp=0xc3d44000, dir=2,
> inp=0x0)
> at /usr/src/sys/contrib/pf/net/pf_ioctl.c:3739
> #8 0xc0657618 in pfil_run_hooks (ph=0xc097ad00, mp=0xe6846638, ifp=0xc3d44000,
> dir=2, inp=0x0)
> at /usr/src/sys/net/pfil.c:78
> #9 0xc07034fe in ip6_output (m0=0xc3c8e900, opt=0x0, ro=0xe6846618,
> flags=Variable "flags" is not available.
> ) at /usr/src/sys/netinet6/ip6_output.c:853
> #10 0xc0477dad in pf_send_tcp (replyto=0xc4fcfe00, r=0xc41259b4, af=28 '\034',
> saddr=0xc4fcfe5c, daddr=0xc4fcfe4c,
> sport=20480, dport=46591, seq=0, ack=1170313007, flags=20 '\024', win=0,
> mss=0, ttl=0 '\0', tag=1, rtag=0, eh=0x0,
> ifp=0xc3f06400) at /usr/src/sys/contrib/pf/net/pf.c:1978
> #11 0xc047e90f in pf_test_tcp (rm=0xe68468e8, sm=0xe68468e4, direction=2,
> kif=0xc3f20900, m=0xc4fcfe00, off=40,
> h=0xc4fcfe44, pd=0xe6846880, am=0xe68468ec, rsm=0xe68468e0, ifq=0x0,
> inp=0xc446b7bc)
> at /usr/src/sys/contrib/pf/net/pf.c:3424
> #12 0xc04816c0 in pf_test6 (dir=2, ifp=0xc3f06400, m0=0xe6846944, eh=0x0,
> inp=0xc446b7bc)
> at /usr/src/sys/contrib/pf/net/pf.c:7368
> #13 0xc0484e37 in pf_check6_out (arg=0x0, m=0xe6846944, ifp=0xc3f06400, dir=2,
> inp=0xc446b7bc)
> at /usr/src/sys/contrib/pf/net/pf_ioctl.c:3739
> #14 0xc0657618 in pfil_run_hooks (ph=0xc097ad00, mp=0xe6846adc, ifp=0xc3f06400,
> dir=2, inp=0xc446b7bc)
> at /usr/src/sys/net/pfil.c:78
> #15 0xc07034fe in ip6_output (m0=0xc4fcfe00, opt=0x0, ro=0xe6846abc,
> flags=Variable "flags" is not available.
> ) at /usr/src/sys/netinet6/ip6_output.c:853
> #16 0xc06debbe in tcp_output (tp=0xc45553a0) at
> /usr/src/sys/netinet/tcp_output.c:1114
> #17 0xc06ea5d1 in tcp6_usr_connect (so=0xc50cd340, nam=0xc447e7c0,
> td=0xc4eed690) at tcp_offload.h:257
> #18 0xc060b002 in soconnect (so=0xc50cd340, nam=0xc447e7c0, td=0xc4eed690) at
> /usr/src/sys/kern/uipc_socket.c:771
> #19 0xc06129e9 in kern_connect (td=0xc4eed690, fd=3, sa=0xc447e7c0) at
> /usr/src/sys/kern/uipc_syscalls.c:570
> #20 0xc0612b56 in connect (td=0xc4eed690, uap=0xe6846cfc) at
> /usr/src/sys/kern/uipc_syscalls.c:534
> #21 0xc083a2d4 in syscall (frame=0xe6846d38) at
> /usr/src/sys/i386/i386/trap.c:1090
> #22 0xc0821220 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
> #23 0x00000033 in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> (kgdb)
>
> >How-To-Repeat:
>
> >Fix:
>
>
>
> >Release-Note:
> >Audit-Trail:
> >Unformatted:
> _______________________________________________
> freebsd-bugs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
> To unsubscribe, send any mail to "freebsd-bugs-unsubscribe at freebsd.org"
More information about the freebsd-pf
mailing list