misc/166340: Process under FreeBSD 9.0 hangs in uninterruptable sleep with apparently no syscall (empty wchan)

[Christian Esken]

>Number:         166340
>Category:       misc
>Synopsis:       Process under FreeBSD 9.0 hangs in uninterruptable sleep with apparently no syscall (empty wchan)
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Mar 23 16:20:11 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     Christian Esken
>Release:        FreeBSD 9.0
>Organization:
>Environment:
FreeBSD dev 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Thu Mar  1 17:17:19 CET 2012     root at dev:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
I have a process that sometimes goes into uninterruptable sleep under FreeBSD 9.0. "ps" shows the process is in "D" state and wchan is "-". The process works reliably under various other operating systems, including misc FreeBSD 8 versions.

I presume it should never happen that a process is in uninterruptible sleep without showing the kernel call in wchan. What could be the reason, or how can I further tackle the situation?

The nature of the process is a remote logger. It reads access logs from a pipe/STDIN (Apache httpd CustomLog) and sends them via TCP to a "scribed" logging server. More details can be found on Stackoverflow [1].

Below is some more information, like "procstat -k" and uname output.

---------------

This is the result from "procstat -k" and  "procstat -kk"

# procstat -k 38013
  PID    TID COMM             TDNAME           KSTACK                       
38013 100817 serelog          -                mi_switch sleepq_check_timeout sleepq_timedwait_sig _sleep soreceive_generic kern_recvit recvit sys_recvfrom amd64_syscall Xfast_syscall 

# procstat -kk 38013
  PID    TID COMM             TDNAME           KSTACK                       
38013 100817 serelog          -                mi_switch+0x174 sleepq_check_timeout+0x80 sleepq_timedwait_sig+0x20 _sleep+0x1b1 soreceive_generic+0xf95 kern_recvit+0x205 recvit+0x21 sys_recvfrom+0x82 amd64_syscall+0x450 Xfast_syscall+0xf7 

---------------

uname -a:
FreeBSD dev 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Thu Mar  1 17:17:19 CET 2012     root at dev:/usr/obj/usr/src/sys/GENERIC  amd64

There is no NFS involved on the system.

Best regards,
     Christian

[1] http://stackoverflow.com/questions/9786501/process-under-freebsd-9-0-hangs-in-uninterruptable-sleep-with-apparently-no-sysc
>How-To-Repeat:
I can reproduce the issue quickly by sending a lot of HTTP requests. The problem does not occur when tracing the process with truss.
If not solvable by just the description, I will isolate a small test case from the full source code.
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: