Processes stuck in *unp_mtx state
Sergey Aleynikov
sergey.aleynikov at gmail.com
Wed Oct 21 10:59:08 UTC 2009
I'm running FreeBSD 7.1-RELEASE and have application that spawnes some
children and (heavily) communicates between them using pipes. But
under load parent and one child both get stuck in unp_mtx state, they
can't be killed (killing them puts whole system on hang). This is
reproducable (i've written client emulation that causes this situation
after ~5 min run). Everything else keeps running, but trying to access
these stuck porcesses puts whole system on hang, so i have to perform
reboot. Panic is not invoked, so i've set up ddb panic autocollecting
script and had manually put system into panic.
Is this a known issue in 7.1, so upgrade to 7.2 would help? I list
below some info about system+ddb output:
begom% uname -a
FreeBSD begom.com 7.1-RELEASE FreeBSD 7.1-RELEASE #1: Mon Sep 14
07:37:56 MSD 2009
inferno at begom.com:/usr/obj/usr/src/release-7.1/sys/BEGOM i386
db:0:kdb.enter.panic> show pcpu
cpuid = 1
curthread = 0x84d00d20: pid 1 "init"
curpcb = 0xa940bd90
fpcurthread = none
idlethread = 0x84d00af0: pid 11 "idle: cpu1"
APIC ID = 1
currentldt = 0x50
db:0:kdb.enter.panic> show allpcpu
Current CPU: 1
cpuid = 0
curthread = 0x850e08c0: pid 1078 "mysqld"
curpcb = 0xae5d4d90
fpcurthread = none
idlethread = 0x84d008c0: pid 12 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
cpuid = 1
curthread = 0x84d00d20: pid 1 "init"
curpcb = 0xa940bd90
fpcurthread = none
idlethread = 0x84d00af0: pid 11 "idle: cpu1"
APIC ID = 1
currentldt = 0x50
db:0:kdb.enter.panic> ps
pid ppid pgrp uid state wmesg wchan cmd
...
89782 89764 89764 1013 L *unp_mtx 0x850b9b40 perl5.8.8
89764 1 89764 1013 Ls *unp_mtx 0x856a70f0 perl5.8.8
db:0:kdb.enter.panic> alltrace
...
Tracing command perl5.8.8 pid 89764 tid 100434 td 0x85672690
sched_switch(85672690,0,1,23f76982,201f8b,...) at sched_switch+0x43b
mi_switch(1,0,80831849,2d9,85bf2000,...) at mi_switch+0x146
turnstile_wait(856a70f0,85bf2000,0,8675d690,87087690,...) at
turnstile_wait+0x2ec
_mtx_lock_sleep(8675d720,85672690,0,0,0,...) at _mtx_lock_sleep+0x10e
uipc_peeraddr(8bb431a0,ae6cac70,85672690,ae6cac80,86b74b94,...) at
uipc_peeraddr+0xc5
kern_getpeername(85672690,11,ae6cac70,ae6cac6c,100,...) at kern_getpeername+0x60
getpeername(85672690,ae6cacfc,c,16,ae6cad2c,...) at getpeername+0x52
syscall(ae6cad38) at syscall+0x335
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (31, FreeBSD ELF32, getpeername), eip = 0x282b88db, esp =
0x7fbfeb5c, ebp = 0x7fbfeba8 ---
Tracing command perl5.8.8 pid 89782 tid 100248 td 0x85bf2000
sched_switch(85bf2000,0,1,23f73c9c,201f8b,...) at sched_switch+0x43b
mi_switch(1,0,80831849,2d9,805d7233,...) at mi_switch+0x146
turnstile_wait(850b9b40,85672690,0,87087690,8675d690,...) at
turnstile_wait+0x2ec
_mtx_lock_sleep(87087720,85bf2000,0,0,0,...) at _mtx_lock_sleep+0x10e
uipc_peeraddr(8f1a8d00,ae3f4c70,ae3f4c44,4aded50c,84ff5da8,...) at
uipc_peeraddr+0xc5
kern_getpeername(85bf2000,6,ae3f4c70,ae3f4c6c,100,...) at kern_getpeername+0x60
getpeername(85bf2000,ae3f4cfc,c,16,ae3f4d2c,...) at getpeername+0x52
syscall(ae3f4d38) at syscall+0x335
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (31, FreeBSD ELF32, getpeername), eip = 0x282b88db, esp =
0x7fbfeb5c, ebp = 0x7fbfeba8 ---
db:0:kdb.enter.panic> show locks
No such command
db:0:kdb.enter.panic> show alllocks
No such command
db:0:kdb.enter.panic> show lockedvnods
Locked vnodes
options CONFIG_AUTOGENERATED
ident BEGOM
machine i386
cpu I686_CPU
makeoptions DEBUG=-g
options ATA_STATIC_ID
options SMP
options AUDIT
options STOP_NMI
options ADAPTIVE_GIANT
options KBD_INSTALL_CDEV
options _KPOSIX_PRIORITY_SCHEDULING
options SYSVSEM
options SYSVMSG
options SYSVSHM
options KTRACE
options SCSI_DELAY=5000
options COMPAT_FREEBSD6
options COMPAT_FREEBSD5
options COMPAT_FREEBSD4
options COMPAT_43TTY
options GEOM_LABEL
options GEOM_PART_GPT
options PSEUDOFS
options PROCFS
options CD9660
options MD_ROOT
options UFS_GJOURNAL
options UFS_DIRHASH
options UFS_ACL
options SOFTUPDATES
options FFS
options INET
options PREEMPTION
options SCHED_ULE
options KVA_PAGES=512
options DEVICE_POLLING
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPFIREWALL
options INCLUDE_CONFIG_FILE
options ALT_BREAK_TO_DEBUGGER
options KDB_UNATTENDED
options DDB
options KDB
options KSE
options GEOM_MBR
options GEOM_BSD
options ISAPNP
Loaded kernel modules are: linux, acpi, aio (but i've seen this both
with and without aio).
(part from fstat)
game perl5.8.8 89764 root / 2 drwxr-xr-x 512 r
game perl5.8.8 89764 wd /home 15191040 drwxr-x--x 3072 r
game perl5.8.8 89764 text /usr 615835 -rwxr-xr-x 29935 r
game perl5.8.8 89764 0 /dev 117 crw--w---- ttyp1 rw
game perl5.8.8 89764 1 /dev 117 crw--w---- ttyp1 rw
game perl5.8.8 89764 2 /dev 117 crw--w---- ttyp1 rw
game perl5.8.8 89764 3* local dgram 86a8e3f0 <-> 852dc0a8
game perl5.8.8 89764 4 /var 329768 -rw-r--r-- 6 rw
game perl5.8.8 89764 5 /usr 733593 -r--r--r-- 6611 r
game perl5.8.8 89764 7* local stream 8678cc78 <-> 8579bc78
game perl5.8.8 89764 8* local stream 8579b000 <-> 8579bbd0
game perl5.8.8 89764 9* local stream 8678c000 <-> 86e07150
game perl5.8.8 89764 10* local stream 86e07888 <-> 86e07d20
game perl5.8.8 89764 11* local stream 865ca0a8 <-> 875d5a80
game perl5.8.8 89764 12* local stream 8678c0a8 <-> 865ca738
game perl5.8.8 89764 13* local stream 8736b540 <-> 8678c690
game perl5.8.8 89764 14* local stream 8579bdc8 <-> 8736bbd0
game perl5.8.8 89764 15* local stream 85a5a498 <-> 875d5498
game perl5.8.8 89764 16* local stream 85c2a3f0 <-> 85c2a348
game perl5.8.8 89764 17* local stream 87087690 <-> 8675d690
game perl5.8.8 89764 18* local stream 85a5a888 <-> 865ca2a0
game perl5.8.8 89764 19* local stream 852c79d8 <-> 8528e3f0
game perl5.8.8 89764 20* internet stream tcp 8d7d9570
game perl5.8.8 89764 21* internet stream tcp 85e021d0
game perl5.8.8 89764 22* internet stream tcp 899f71d0
game perl5.8.8 89764 23* internet stream tcp 8e8bd1d0
game perl5.8.8 89764 24* internet stream tcp 8c137000
game perl5.8.8 89764 25* internet stream tcp 8d39b3a0
game perl5.8.8 89764 26* internet stream tcp 85bc0000
game perl5.8.8 89764 27* internet stream tcp 8c78e570
game perl5.8.8 89764 28* internet stream tcp 8a7fa910
game perl5.8.8 89764 29* internet stream tcp 8dd6bcb0
game perl5.8.8 89764 30* internet stream tcp 8e801570
game perl5.8.8 89764 31* internet stream tcp 86635cb0
game perl5.8.8 89764 32* internet stream tcp 8cd50740
game perl5.8.8 89764 33* internet stream tcp 86656000
More information about the freebsd-stable
mailing list