Process hung in STOPPED_SINGLE, wchan vodead, and cannot be killed or continued

[Palle Girgensohn]

Hi,

I see a process that is hung in a jail, and cannot be killed or continued:

# ps HO wchan,nwchan,ppid -p 92266
  PID WCHAN  NWCHAN           PPID TT  STAT    TIME COMMAND
92266 -      -                   1  -  TJ   0:00,73 /usr/local/bin/jsvc -home /usr/local/openjdk8 -server
92266 vodead fffff811a5e6b400    1  -  TJ   0:00,48 /usr/local/bin/jsvc -home /usr/local/openjdk8 -server

# top
...
  PID USERNAME     THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
92266 nobody         2  20    0  4470M   418M STOP    2   0:20   0.00% jsvc


# ps axu
USER     PID %CPU %MEM     VSZ    RSS TT  STAT STARTED    TIME COMMAND
nobody 92266  0,0  0,4 4577204 427756  -  TJ   11:02pm 0:20,08 /usr/local/bin/jsvc -home /usr/local/openjdk8 ...

# sockstat
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
nobody   jsvc       92266 15 stream (not connected)
nobody   jsvc       92266 16 tcp4   127.0.0.1:8078        *:*
?        ?          ?     ?  tcp4   127.0.0.1:8078        127.0.0.1:22789
...

# sockstat | grep '^?' |wc -l
     151

# netstat -an | less
netstat: kvm not available: /dev/mem: No such file or directory
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4     374      0 127.0.0.1.8078         127.0.0.1.32866        CLOSED
...


# procstat -t 92266
  PID    TID COMM             TDNAME           CPU  PRI STATE   WCHAN    
92266 105754 jsvc             -                 20  120 stop    -         
92266 106982 jsvc             -                  2  120 stop    vodead    
# procstat -k 92266
  PID    TID COMM             TDNAME           KSTACK                       
92266 105754 jsvc             -                mi_switch thread_suspend_switch thread_single exit1 sys_sys_exit amd64_syscall Xfast_syscall 
92266 106982 jsvc             -                mi_switch sleepq_switch sleepq_wait _sleep vnode_create_vobject zfs_freebsd_open VOP_OPEN_APV vn_open_vnode vn_open_cred kern_openat amd64_syscall Xfast_syscall 



8078 is the java port that it used to listen to...


all look like this
?        ?          ?     ?  tcp4   127.0.0.1:8078        127.0.0.1:53583


# gdb -p 92266  /usr/local/bin/jsvc 
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)...
Attaching to program: /usr/local/bin/jsvc, process 92266

[ just hangs ]...

^Z
[1]+  Stopped                 gdb -p 92266 /usr/local/bin/jsvc
[root at tranbar /]# 
[root at tranbar /]# 
[root at tranbar /]# kill %1
[root at tranbar /]# 
[1]+  Terminated              gdb -p 92266 /usr/local/bin/jsvc
[root at tranbar /]# 




The culprit to begin with could be this:

Oct  7 07:54:00 host kernel: sonewconn: pcb 0xfffff80b49171310: Listen queue overflow: 151 already in queue awaiting acceptance (6 occurrences)

Occurred all through the night, saturating a service, *very likely* the one now showing problems, but i was never there to check. 151 lost network sockets (see sockstat above) connects the dots.

It seems the service entered STOP when we tried to stop it. jsvc is similar to daemontools, and I remeber seeing a references to a parent process 92265, but I might be imaginating, since the ppid = 1.

Trying to shut down the jail we got hanging shutdown processes:

from host:/var/log/console.jailname:
...
Stopping tomcat.
Waiting for PIDS: 9226690 second watchdog timeout expired. Shutdown terminated.
Ons  7 Okt 2015 08:27:19 CEST
...


# freebsd-version -ku
10.2-RELEASE-p3
10.2-RELEASE-p3


So basically, is there a way to get rid of this process without rebooting?

Palle