an unkillable process and a patch

[Garry Belka]

We see, not too often, that a Java process hangs and can't be killed 
even by SIGKILL.

Apparently, one of the process threads forks. fork1() in kernel attempts 
to enter a single-threaded mode, but thread_single() fails to complete 
and hangs waiting until all threads but proc-> p_singlethread are 
suspended. One of the remaining threads is not suspended and has only 
SLEEP flag set.

pid    thread    thid  flags  inhib pflags  comm         wchan

  1982 0xcd150180 100351 00020c00    1 0088 java 
<sched_switch+323>
     mi_switch + 426 in section .text
     thread_suspend_check + 298 in section .text
     userret + 58 in section .text
     fork_return + 18 in section .text
     fork_exit + 102 in section .text
  1982 0xce120c00 100948 00000c00    1 0880 java 
<sched_switch+323>
     mi_switch + 426 in section .text
     thread_suspend_check + 298 in section .text
     userret + 58 in section .text
     ast + 844 in section .text
  1982 0xcd740900 100616 00000808    2 0080 java         sbwait cd557320
     mi_switch + 426 in section .text              (SLEEPING, not SUSPENDED)
     sleepq_switch + 164 in section .text
     sleepq_wait_sig + 12 in section .text
     msleep + 566 in section .text
     sbwait + 56 in section .text
     soreceive + 572 in section .text
     soo_read + 65 in section .text
     dofileread + 173 in section .text
     read + 59 in section .text
     syscall + 551 in section .text
  1982 0xc3ae7900 100906 00000808    1 0080 java             mi_switch + 
426 in section .text
     sleepq_switch + 164 in section .text
     sleepq_wait_sig + 12 in section .text
     msleep + 566 in section .text
     sbwait + 56 in section .text
     soreceive + 572 in section .text
     soo_read + 65 in section .text
     dofileread + 173 in section .text
     read + 59 in section .text
     syscall + 551 in section .text
  1982 0xcd719780 100605 00000c00    1 0880 java             mi_switch + 
426 in section .text
     thread_suspend_check + 298 in section .text
     userret + 58 in section .text
     ast + 844 in section .text
  1982 0xcd6d9000 100830 00000000    1 0880 java      (p_singlethread) 
      mi_switch + 426 in section .text     - line 355
     thread_single + 497 in section .text - line 863
     fork1 + 169 in section .text         - line 257
     fork + 24 in section .text
     syscall + 551 in section .text

Signals in singlethread state are not really delivered, SIGKILL stays 
with the first thread in the queue, and so we got a deadlock.


I think that we got into this state because the non-suspended thread was 
running when singlethread was attempting to put every thread to sleep. 
All threads were marked TDF_ASTPENDING. However, a bit later ast() 
failed to deal correctly with a thread that had non-null td->td_mailbox.

sys/kern/subr_trap.c:ast()
        if ((p->p_flag & P_SA) && (td->td_mailbox == NULL))
                 thread_user_enter(td);

Below is a tentative patch. It's for 5.4-stable but it seemsa to me that 
the same problem should be in 6.0.

Any comments?

Best,
Garry

Index: kern/kern_thread.c
===================================================================
RCS file: /u1/Repo/FreeBSD/sys/kern/kern_thread.c,v
retrieving revision 1.3
diff -u -r1.3 kern_thread.c
--- kern/kern_thread.c  9 Jul 2005 01:27:18 -0000       1.3
+++ kern/kern_thread.c  15 Nov 2005 03:01:22 -0000
@@ -1001,6 +1001,18 @@
  }

  void
+thread_check_single_suspend(struct thread *td)
+{
+        struct proc *p = td->td_proc;
+
+        if (__predict_false(P_SHOULDSTOP(p))) {
+                PROC_LOCK(p);
+                thread_suspend_check(0);
+                PROC_UNLOCK(p);
+        }
+}
+
+void
  thread_unsuspend_one(struct thread *td)
  {
         struct proc *p = td->td_proc;
Index: kern/subr_trap.c
===================================================================
RCS file: /u1/Repo/FreeBSD/sys/kern/subr_trap.c,v
retrieving revision 1.1.1.2
diff -u -r1.1.1.2 subr_trap.c
--- kern/subr_trap.c    8 Jul 2005 03:01:08 -0000       1.1.1.2
+++ kern/subr_trap.c    15 Nov 2005 03:01:23 -0000
@@ -171,6 +171,8 @@

         if ((p->p_flag & P_SA) && (td->td_mailbox == NULL))
                 thread_user_enter(td);
+        else
+               thread_check_single_suspend(td);
         /*
          * This updates the p_sflag's for the checks below in one
          * "atomic" operation with turning off the astpending flag.
Index: sys/proc.h
===================================================================
RCS file: /u1/Repo/FreeBSD/sys/sys/proc.h,v
retrieving revision 1.1.1.5
diff -u -r1.1.1.5 proc.h
--- sys/proc.h  8 Jul 2005 03:07:51 -0000       1.1.1.5
+++ sys/proc.h  15 Nov 2005 03:01:28 -0000
@@ -887,6 +887,7 @@
  void   ksegrp_unlink(struct ksegrp *kg);
  void   thread_signal_add(struct thread *td, int sig);
  struct thread *thread_alloc(void);
+void   thread_check_single_suspend(struct thread *td);
  void   thread_exit(void) __dead2;
  int    thread_export_context(struct thread *td, int willexit);
  void   thread_free(struct thread *td);