kern/159390: panic: mutex pf task mtx owned at /usr/src/sys/contrib/pf/net/if_pfsync.c:2029

[Victor Detoni]

>Number:         159390
>Category:       kern
>Synopsis:       panic: mutex pf task mtx owned at /usr/src/sys/contrib/pf/net/if_pfsync.c:2029
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Aug 02 18:50:09 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Victor Detoni
>Release:        FreeBSD 9.0-BETA1 - CURRENT
>Organization:
>Environment:
FreeBSD fbsd2 9.0-BETA1 FreeBSD 9.0-BETA1 #0: Thu Jul 28 17:15:31 UTC 2011     root at farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
Problem occured after packets SYN sent to pf interface.

lock order reversal:
 1st 0xffffffff8112bba0 pf task mtx (pf task mtx) @ /usr/src/sys/contrib/pf/net/pf.c:6630
 2nd 0xfffffe000357baf8 radix mode head (radix node head) @ /usr/src/sys/net/route.c:354
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
kdb_backtrace() at _witness_debugger+0x2e
witness_checkorder() at witness_checkorder+0x807
Aug  1 21:20:13 fbsd2 kernel: _rw_rlock() at _rw_rlock+0x6d
Aug  1 21:20:13 fbsd2 kernel: rtalloc1_fib() at rtalloc1_fib+0x10c
Aug  1 21:20:13 fbsd2 kernel: rtalloc_ign_fib() at rtalloc_ign_fib+0xc5
Aug  1 21:20:13 fbsd2 kernel: pf_calc_mss() at pf_calc_mss+0xb2
Aug  1 21:20:13 fbsd2 kernel: pf_test_rule() at pf_test_rule+0x1e74
Aug  1 21:20:13 fbsd2 kernel: pf_test() at pf_test+0x1051
Aug  1 21:20:13 fbsd2 kernel: pf_check_in() at pf_check_in+0x2b
Aug  1 21:20:13 fbsd2 kernel: pfil_run_hooks() at pfil_run_hooks+0xd2
Aug  1 21:20:13 fbsd2 kernel: ip_input() at ip_input+0x2e7
Aug  1 21:20:13 fbsd2 kernel: netisr_dispatch_src() at netisr_dispatch_src+0x160
Aug  1 21:20:13 fbsd2 kernel: ether_demux() at ether_demux+0x17d
Aug  1 21:20:13 fbsd2 kernel: ether_nh_input() at ether_nh_input+0x20e
Aug  1 21:20:13 fbsd2 kernel: netisr_dispatch_src() at netisr_dispatch_src+0x160
Aug  1 21:20:13 fbsd2 kernel: em_rxeof() at em_rxeof+0x1a7
Aug  1 21:20:13 fbsd2 kernel: em_handle_que() at em_handle_que+0x50
Aug  1 21:20:13 fbsd2 kernel: taskqueue_run_locked() at taskqueue_run_locked+0x93
Aug  1 21:20:13 fbsd2 kernel: taskqueue_thread_loop() at taskqueue_thread_loop+0x3e
Aug  1 21:20:13 fbsd2 kernel: fork_exit() at fork_exit+0x135
Aug  1 21:20:13 fbsd2 kernel: fork_trampoline() at fork_trampoline+0xe
panic: mutex pf task mtx owned at /usr/src/sys/contrib/pf/net/if_pfsync.c:2029
cpuid = 0
KDB: enter: panic
[ thread pid 0 tid 100030 ]
Stopped at          kdb_enter+0x3b: movq $0,0x924002(xrip)
db>
db>

>How-To-Repeat:
After reboot, it shows: (panic)

Aug  2 14:58:05 fbsd2 kernel: lock order reversal:
Aug  2 14:58:05 fbsd2 kernel: 1st 0xffffff80f6386258 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2658
Aug  2 14:58:05 fbsd2 kernel: 2nd 0xfffffe00035fc600 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
Aug  2 14:58:05 fbsd2 kernel: KDB: stack backtrace:
Aug  2 14:58:05 fbsd2 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
Aug  2 14:58:05 fbsd2 kernel: kdb_backtrace() at kdb_backtrace+0x37
Aug  2 14:58:05 fbsd2 kernel: _witness_debugger() at _witness_debugger+0x2e
Aug  2 14:58:05 fbsd2 kernel: witness_checkorder() at witness_checkorder+0x807
Aug  2 14:58:05 fbsd2 kernel: _sx_xlock() at _sx_xlock+0x55
Aug  2 14:58:05 fbsd2 kernel: ufsdirhash_acquire() at ufsdirhash_acquire+0x33
Aug  2 14:58:05 fbsd2 kernel: ufsdirhash_add() at ufsdirhash_add+0x19
Aug  2 14:58:05 fbsd2 kernel: ufs_direnter() at ufs_direnter+0x8c9
Aug  2 14:58:05 fbsd2 kernel: ufs_makeinode() at ufs_makeinode+0x25b
Aug  2 14:58:05 fbsd2 kernel: VOP_CREATE_APV() at VOP_CREATE_APV+0x8d
Aug  2 14:58:05 fbsd2 kernel: vn_open_cred() at vn_open_cred+0x46a
Aug  2 14:58:05 fbsd2 kernel: kern_openat() at kern_openat+0x17f
Aug  2 14:58:05 fbsd2 kernel: syscallenter() at syscallenter+0x1aa
Aug  2 14:58:05 fbsd2 kernel: syscall() at syscall+0x4c
Aug  2 14:58:05 fbsd2 kernel: Xfast_syscall() at Xfast_syscall+0xdd
Aug  2 14:58:05 fbsd2 kernel: --- syscall (5, FreeBSD ELF64, open), rip = 0x800b3de2c, rsp = 0x7fffffffd358, rbp = 0x8 ---
Aug  2 14:58:05 fbsd2 kernel: lock order reversal:
Aug  2 14:58:05 fbsd2 kernel: 1st 0xfffffe00b115e638 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2134
Aug  2 14:58:05 fbsd2 kernel: 2nd 0xffffff80f6386258 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:261
Aug  2 14:58:05 fbsd2 kernel: 3rd 0xfffffe002a4e3818 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2134
Aug  2 14:58:05 fbsd2 kernel: KDB: stack backtrace:
Aug  2 14:58:05 fbsd2 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
Aug  2 14:58:05 fbsd2 kernel: kdb_backtrace() at kdb_backtrace+0x37
Aug  2 14:58:05 fbsd2 kernel: _witness_debugger() at _witness_debugger+0x2e
Aug  2 14:58:05 fbsd2 kernel: witness_checkorder() at witness_checkorder+0x807
Aug  2 14:58:05 fbsd2 kernel: __lockmgr_args() at __lockmgr_args+0xd42
Aug  2 14:58:05 fbsd2 kernel: ffs_lock() at ffs_lock+0x8c
Aug  2 14:58:05 fbsd2 kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
Aug  2 14:58:05 fbsd2 kernel: _vn_lock() at _vn_lock+0x47
Aug  2 14:58:05 fbsd2 kernel: vget() at vget+0x7b
Aug  2 14:58:05 fbsd2 kernel: vfs_hash_get() at vfs_hash_get+0xd5
Aug  2 14:58:05 fbsd2 kernel: ffs_vgetf() at ffs_vgetf+0x48
Aug  2 14:58:05 fbsd2 kernel: softdep_sync_buf() at softdep_sync_buf+0x547
Aug  2 14:58:05 fbsd2 kernel: ffs_syncvnode() at ffs_syncvnode+0x299
Aug  2 14:58:05 fbsd2 kernel: ffs_truncate() at ffs_truncate+0x463
Aug  2 14:58:05 fbsd2 kernel: ufs_direnter() at ufs_direnter+0x6ff
Aug  2 14:58:05 fbsd2 kernel: ufs_makeinode() at ufs_makeinode+0x25b
Aug  2 14:58:05 fbsd2 kernel: VOP_CREATE_APV() at VOP_CREATE_APV+0x8d
Aug  2 14:58:05 fbsd2 kernel: vn_open_cred() at vn_open_cred+0x46a
Aug  2 14:58:05 fbsd2 kernel: kern_openat() at kern_openat+0x17f
Aug  2 14:58:05 fbsd2 kernel: syscallenter() at syscallenter+0x1aa
Aug  2 14:58:05 fbsd2 kernel: syscall() at syscall+0x4c
Aug  2 14:58:05 fbsd2 kernel: Xfast_syscall() at Xfast_syscall+0xdd
Aug  2 14:58:05 fbsd2 kernel: --- syscall (5, FreeBSD ELF64, open), rip = 0x800b3de2c, rsp = 0x7fffffffd358, rbp = 0x8 ---
panic: mutex pf task mtx owned at /usr/src/sys/contrib/pf/net/if_pfsync.c:2029
cpuid = 0
KDB: enter: panic
[ thread pid 0 tid 100030 ]
Stopped at          kdb_enter+0x3b: movq $0,0x924002(xrip)
db>

>Fix:
Recompile kernel without "device pfsync" line.

>Release-Note:
>Audit-Trail:
>Unformatted: