[Bug 186114] net/mpd5 hangs after a certain number of users connect

bugzilla-noreply at freebsd.org bugzilla-noreply at freebsd.org
Tue Jun 27 21:14:11 UTC 2017 

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=186114

--- Comment #83 from Cassiano Peixoto <peixoto.cassiano at gmail.com> ---
(In reply to Eugene Grosbein from comment #81)
Eugene and Konstantin,

Bad news, it just stopped working. Eugene i hadn't enabled web server yet. So
it stucked the mpd5 process with patch applied listed on comment #67.

The top out i can see same message:

1295 root             9  52    0   314M 54820K uwrlck  7 625:46   0.74% mpd5

Here it is the bt:

(gdb) bt
#0  0x000000080228567c in _umtx_op_err () from /lib/libthr.so.3
#1  0x0000000802279c81 in __thr_rwlock_wrlock (rwlock=0x802494500, tsp=<value
optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:325
#2  0x00000008022829f3 in _thr_rtld_wlock_acquire (lock=<value optimized out>)
at thr_umtx.h:239
#3  0x00000008006aec72 in wlock_acquire (lock=0x8008c3960,
lockstate=0x7fffdfffda08) at /usr/src/libexec/rtld-elf/rtld_lock.c:217
#4  0x00000008006aaf4f in _rtld_allocate_tls (oldtls=0x0, tcbsize=32,
tcbalign=16) at /usr/src/libexec/rtld-elf/rtld.c:4802
#5  0x0000000802281b69 in _tcb_ctor (thread=0x80c740f00, initial=<value
optimized out>) at /usr/src/lib/libthr/thread/thr_ctrdtr.c:45
#6  0x0000000802281110 in _thr_alloc (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_list.c:170
#7  0x00000008022772a6 in _pthread_create (thread=0x80324d598, attr=0x0,
start_routine=0x4653b0 <fseeko at plt+388260>, arg=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:78
#8  0x000000000046535b in ?? ()
#9  0x00000000004292a8 in ?? ()
#10 0x000000000042d219 in ?? ()
#11 0x0000000000448da3 in ?? ()
#12 0x00000000004493b1 in ?? ()
#13 0x000000000043ac94 in ?? ()
#14 0x000000000043d650 in ?? ()
#15 0x000000000043d366 in ?? ()
#16 0x000000000043b21d in ?? ()
#17 0x0000000000445012 in ?? ()
#18 0x000000000044b598 in ?? ()
#19 0x0000000000439c9f in ?? ()
#20 0x00000000004651d6 in ?? ()
#21 0x0000000000464908 in ?? ()
#22 0x0000000802277b55 in thread_start (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:289

Here it is all threads:

(gdb) thread apply all bt

Thread 9 (Thread 803016a00 (LWP 100704/<unknown>)):
#0  0x000000080228567c in _umtx_op_err () from /lib/libthr.so.3
#1  0x0000000802279c81 in __thr_rwlock_wrlock (rwlock=0x802494500, tsp=<value
optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:325
#2  0x00000008022829f3 in _thr_rtld_wlock_acquire (lock=<value optimized out>)
at thr_umtx.h:239
#3  0x00000008006aec72 in wlock_acquire (lock=0x8008c3960,
lockstate=0x7fffdfffda08) at /usr/src/libexec/rtld-elf/rtld_lock.c:217
#4  0x00000008006aaf4f in _rtld_allocate_tls (oldtls=0x0, tcbsize=32,
tcbalign=16) at /usr/src/libexec/rtld-elf/rtld.c:4802
#5  0x0000000802281b69 in _tcb_ctor (thread=0x80c740f00, initial=<value
optimized out>) at /usr/src/lib/libthr/thread/thr_ctrdtr.c:45
#6  0x0000000802281110 in _thr_alloc (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_list.c:170
#7  0x00000008022772a6 in _pthread_create (thread=0x80324d598, attr=0x0,
start_routine=0x4653b0 <fseeko at plt+388260>, arg=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:78
#8  0x000000000046535b in ?? ()
#9  0x00000000004292a8 in ?? ()
#10 0x000000000042d219 in ?? ()
#11 0x0000000000448da3 in ?? ()
#12 0x00000000004493b1 in ?? ()
#13 0x000000000043ac94 in ?? ()
#14 0x000000000043d650 in ?? ()
#15 0x000000000043d366 in ?? ()
#16 0x000000000043b21d in ?? ()
#17 0x0000000000445012 in ?? ()
#18 0x000000000044b598 in ?? ()
#19 0x0000000000439c9f in ?? ()
#20 0x00000000004651d6 in ?? ()
#21 0x0000000000464908 in ?? ()
#22 0x0000000802277b55 in thread_start (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:289
#23 0x0000000000000000 in ?? ()

Thread 8 (Thread 80326ca00 (LWP 101005/<unknown>)):
#0  0x000000080257d43a in _nanosleep () from /lib/libc.so.7
#1  0x000000080227a4cc in __thr_nanosleep (time_to_sleep=0x7fffc6731ed0,
time_remaining=0x0) at /usr/src/lib/libthr/thread/thr_syscalls.c:257
#2  0x00000008025d5076 in __usleep (useconds=<value optimized out>) at
/usr/src/lib/libc/gen/usleep.c:52
#3  0x0000000000465532 in ?? ()
#4  0x0000000802285550 in __pthread_cleanup_pop_imp (execute=<value optimized
out>) at /usr/src/lib/libthr/thread/thr_clean.c:73
#5  0x000000000046541e in ?? ()
#6  0x0000000802277b55 in thread_start (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:289
#7  0x00007fffc6532000 in ?? ()
Cannot access memory at address 0x7fffc6732000

Thread 7 (Thread 80326ed00 (LWP 101007/<unknown>)):
#0  0x000000080257d43a in _nanosleep () from /lib/libc.so.7
#1  0x000000080227a4cc in __thr_nanosleep (time_to_sleep=0x7fffb7cbced0,
time_remaining=0x0) at /usr/src/lib/libthr/thread/thr_syscalls.c:257
#2  0x00000008025d5076 in __usleep (useconds=<value optimized out>) at
/usr/src/lib/libc/gen/usleep.c:52
#3  0x0000000000465532 in ?? ()
#4  0x0000000802285550 in __pthread_cleanup_pop_imp (execute=<value optimized
out>) at /usr/src/lib/libthr/thread/thr_clean.c:73
#5  0x000000000046541e in ?? ()
#6  0x0000000802277b55 in thread_start (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:289
#7  0x00007fffb7abd000 in ?? ()
Cannot access memory at address 0x7fffb7cbd000

---Type <return> to continue, or q <return> to quit---
Thread 6 (Thread 803252e00 (LWP 100809/<unknown>)):
#0  0x000000080257d43a in _nanosleep () from /lib/libc.so.7
#1  0x000000080227a4cc in __thr_nanosleep (time_to_sleep=0x7fffb86c1ed0,
time_remaining=0x0) at /usr/src/lib/libthr/thread/thr_syscalls.c:257
#2  0x00000008025d5076 in __usleep (useconds=<value optimized out>) at
/usr/src/lib/libc/gen/usleep.c:52
#3  0x0000000000465532 in ?? ()
#4  0x0000000802285550 in __pthread_cleanup_pop_imp (execute=<value optimized
out>) at /usr/src/lib/libthr/thread/thr_clean.c:73
#5  0x000000000046541e in ?? ()
#6  0x0000000802277b55 in thread_start (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:289
#7  0x00007fffb84c2000 in ?? ()
Cannot access memory at address 0x7fffb86c2000

Thread 5 (Thread 807292a00 (LWP 100290/<unknown>)):
#0  0x000000080257d43a in _nanosleep () from /lib/libc.so.7
#1  0x000000080227a4cc in __thr_nanosleep (time_to_sleep=0x7fff88b44ed0,
time_remaining=0x0) at /usr/src/lib/libthr/thread/thr_syscalls.c:257
#2  0x00000008025d5076 in __usleep (useconds=<value optimized out>) at
/usr/src/lib/libc/gen/usleep.c:52
#3  0x0000000000465532 in ?? ()
#4  0x0000000802285550 in __pthread_cleanup_pop_imp (execute=<value optimized
out>) at /usr/src/lib/libthr/thread/thr_clean.c:73
#5  0x000000000046541e in ?? ()
#6  0x0000000802277b55 in thread_start (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:289
#7  0x00007fff88945000 in ?? ()
Cannot access memory at address 0x7fff88b45000

Thread 4 (Thread 803253d00 (LWP 101008/<unknown>)):
#0  0x000000080257d43a in _nanosleep () from /lib/libc.so.7
#1  0x000000080227a4cc in __thr_nanosleep (time_to_sleep=0x7fffbdaebed0,
time_remaining=0x0) at /usr/src/lib/libthr/thread/thr_syscalls.c:257
#2  0x00000008025d5076 in __usleep (useconds=<value optimized out>) at
/usr/src/lib/libc/gen/usleep.c:52
#3  0x0000000000465532 in ?? ()
#4  0x0000000802285550 in __pthread_cleanup_pop_imp (execute=<value optimized
out>) at /usr/src/lib/libthr/thread/thr_clean.c:73
#5  0x000000000046541e in ?? ()
#6  0x0000000802277b55 in thread_start (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:289
#7  0x00007fffbd8ec000 in ?? ()
Cannot access memory at address 0x7fffbdaec000

Thread 3 (Thread 808089000 (LWP 101010/<unknown>)):
#0  0x000000080257d43a in _nanosleep () from /lib/libc.so.7
#1  0x000000080227a4cc in __thr_nanosleep (time_to_sleep=0x7fff8bf5eed0,
time_remaining=0x0) at /usr/src/lib/libthr/thread/thr_syscalls.c:257
#2  0x00000008025d5076 in __usleep (useconds=<value optimized out>) at
/usr/src/lib/libc/gen/usleep.c:52
#3  0x0000000000465532 in ?? ()
#4  0x0000000802285550 in __pthread_cleanup_pop_imp (execute=<value optimized
out>) at /usr/src/lib/libthr/thread/thr_clean.c:73
#5  0x000000000046541e in ?? ()
#6  0x0000000802277b55 in thread_start (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:289
#7  0x00007fff8bd5f000 in ?? ()
Cannot access memory at address 0x7fff8bf5f000

Thread 2 (Thread 80808a400 (LWP 101011/<unknown>)):
#0  0x000000080257d43a in _nanosleep () from /lib/libc.so.7
#1  0x000000080227a4cc in __thr_nanosleep (time_to_sleep=0x7fff9178aed0,
time_remaining=0x0) at /usr/src/lib/libthr/thread/thr_syscalls.c:257
#2  0x00000008025d5076 in __usleep (useconds=<value optimized out>) at
/usr/src/lib/libc/gen/usleep.c:52
#3  0x0000000000465532 in ?? ()
#4  0x0000000802285550 in __pthread_cleanup_pop_imp (execute=<value optimized
out>) at /usr/src/lib/libthr/thread/thr_clean.c:73
#5  0x000000000046541e in ?? ()
---Type <return> to continue, or q <return> to quit---
#6  0x0000000802277b55 in thread_start (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:289
#7  0x00007fff9158b000 in ?? ()
Cannot access memory at address 0x7fff9178b000

Thread 1 (Thread 807258d00 (LWP 101012/<unknown>)):
#0  0x000000080257d43a in _nanosleep () from /lib/libc.so.7
#1  0x000000080227a4cc in __thr_nanosleep (time_to_sleep=0x7fff7c2e0ed0,
time_remaining=0x0) at /usr/src/lib/libthr/thread/thr_syscalls.c:257
#2  0x00000008025d5076 in __usleep (useconds=<value optimized out>) at
/usr/src/lib/libc/gen/usleep.c:52
#3  0x0000000000465532 in ?? ()
#4  0x0000000802285550 in __pthread_cleanup_pop_imp (execute=<value optimized
out>) at /usr/src/lib/libthr/thread/thr_clean.c:73
#5  0x000000000046541e in ?? ()
#6  0x0000000802277b55 in thread_start (curthread=<value optimized out>) at
/usr/src/lib/libthr/thread/thr_create.c:289
#7  0x00007fff7c0e1000 in ?? ()
Cannot access memory at address 0x7fff7c2e1000
#0  0x000000080228567c in _umtx_op_err () from /lib/libthr.so.3

Printof of *rwlock:

(gdb) frame 1
#1  0x0000000802279c81 in __thr_rwlock_wrlock (rwlock=0x802494500, tsp=<value
optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:325
325             return (_umtx_op_err(rwlock, UMTX_OP_RW_WRLOCK, 0, (void
*)tm_size,
Current language:  auto; currently minimal
(gdb) p *rwlock
$1 = {rw_state = -1610612736, rw_flags = 2, rw_blocked_readers = 0,
rw_blocked_writers = 0, rw_wowner = 100704, rw_spare = 0x802494514}

I could see some kernel messages as well, earlier today:

Jun 24 10:14:21 B-ras kernel: node: ID [20498d]: type 'tee', 0 hooks, flags
0x9, 0 refs, mpd1295-vlan340-44-lt:
Jun 24 10:14:21 B-ras kernel: Last active @ /usr/src/sys/netgraph/ng_base.c,
line 804
Jun 24 10:14:21 B-ras kernel: problem discovered at file
/usr/src/sys/netgraph/ng_base.c, line 3709
Jun 24 10:14:21 B-ras kernel: KDB: stack backtrace:
Jun 24 10:14:21 B-ras kernel: #0 0xffffffff80b1af87 at kdb_backtrace+0x67
Jun 24 10:14:21 B-ras kernel: #1 0xffffffff80c46671 at ng_send_fn1+0xc1
Jun 24 10:14:21 B-ras kernel: #2 0xffffffff80c3e9c4 at ng_destroy_hook+0x334
Jun 24 10:14:21 B-ras kernel: #3 0xffffffff80c4289b at ng_apply_item+0x3eb
Jun 24 10:14:21 B-ras kernel: #4 0xffffffff80c48475 at ngthread+0x315
Jun 24 10:14:21 B-ras kernel: #5 0xffffffff80a88c55 at fork_exit+0x85
Jun 24 10:14:21 B-ras kernel: #6 0xffffffff80ec5f8e at fork_trampoline+0xe
Jun 24 10:14:21 B-ras kernel: Accessing freed node: ID [20498d]: type 'tee', 0
hooks, flags 0x9, 1 refs, mpd1295-vlan340-44-lt:
Jun 24 10:14:21 B-ras kernel: Last active @ /usr/src/sys/netgraph/ng_base.c,
line 3709
Jun 24 10:14:21 B-ras kernel: problem discovered at file
/usr/src/sys/netgraph/ng_base.c, line 2455
Jun 24 10:14:21 B-ras kernel: KDB: stack backtrace:
Jun 24 10:14:21 B-ras kernel: #0 0xffffffff80b1af87 at kdb_backtrace+0x67
Jun 24 10:14:21 B-ras kernel: #1 0xffffffff80c42711 at ng_apply_item+0x261
Jun 24 10:14:21 B-ras kernel: #2 0xffffffff80c4217e at ng_snd_item+0x1ee
Jun 24 10:14:21 B-ras kernel: #3 0xffffffff80c3e9c4 at ng_destroy_hook+0x334
Jun 24 10:14:21 B-ras kernel: #4 0xffffffff80c4289b at ng_apply_item+0x3eb
Jun 24 10:14:21 B-ras kernel: #5 0xffffffff80c48475 at ngthread+0x315
Jun 24 10:14:21 B-ras kernel: #6 0xffffffff80a88c55 at fork_exit+0x85
Jun 24 10:14:21 B-ras kernel: #7 0xffffffff80ec5f8e at fork_trampoline+0xe
Jun 24 10:14:21 B-ras kernel: Accessing freed node: ID [20498d]: type 'tee', 0
hooks, flags 0x9, 1 refs, mpd1295-vlan340-44-lt:
Jun 24 10:14:21 B-ras kernel: Last active @ /usr/src/sys/netgraph/ng_base.c,
line 2455
Jun 24 10:14:22 B-ras kernel: problem discovered at file
/usr/src/sys/netgraph/ng_base.c, line 2335
Jun 24 10:14:22 B-ras kernel: KDB: stack backtrace:
Jun 24 10:14:22 B-ras kernel: #0 0xffffffff80b1af87 at kdb_backtrace+0x67
Jun 24 10:14:22 B-ras kernel: #1 0xffffffff80c42308 at ng_snd_item+0x378
Jun 24 10:14:22 B-ras kernel: #2 0xffffffff80c3e9c4 at ng_destroy_hook+0x334
Jun 24 10:14:22 B-ras kernel: #3 0xffffffff80c4289b at ng_apply_item+0x3eb
Jun 24 10:14:22 B-ras kernel: #4 0xffffffff80c48475 at ngthread+0x315
Jun 24 10:14:22 B-ras kernel: #5 0xffffffff80a88c55 at fork_exit+0x85
Jun 24 10:14:22 B-ras kernel: #6 0xffffffff80ec5f8e at fork_trampoline+0xe
Jun 24 10:14:22 B-ras kernel: Accessing freed node: ID [20498d]: type 'tee', 0
hooks, flags 0x9, 0 refs, mpd1295-vlan340-44-lt:
Jun 24 10:14:22 B-ras kernel: Last active @ /usr/src/sys/netgraph/ng_base.c,
line 2335
Jun 24 10:14:22 B-ras kernel: problem discovered at file
/usr/src/sys/netgraph/ng_base.c, line 804
Jun 24 10:14:22 B-ras kernel: KDB: stack backtrace:
Jun 24 10:14:22 B-ras kernel: #0 0xffffffff80b1af87 at kdb_backtrace+0x67
Jun 24 10:14:22 B-ras kernel: #1 0xffffffff80c3ee46 at ng_unref_node+0x146
Jun 24 10:14:22 B-ras kernel: #2 0xffffffff80c42326 at ng_snd_item+0x396
Jun 24 10:14:22 B-ras kernel: #3 0xffffffff80c3e9c4 at ng_destroy_hook+0x334
Jun 24 10:14:22 B-ras kernel: #4 0xffffffff80c4289b at ng_apply_item+0x3eb
Jun 24 10:14:22 B-ras kernel: #5 0xffffffff80c48475 at ngthread+0x315
Jun 24 10:14:22 B-ras kernel: #6 0xffffffff80a88c55 at fork_exit+0x85
Jun 24 10:14:22 B-ras kernel: #7 0xffffffff80ec5f8e at fork_trampoline+0xe

Let me know if you need something else.

Thanks.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

More information about the freebsd-net mailing list