Massive performance loss from OS::sleep hack

[Kurt Miller]

On Saturday 15 September 2007 01:50:13 pm Kris Kennaway wrote:
> Hi,
>
> I have been running the volano java benchmark
> (http://www.volano.com/benchmarks.html) on an 8-core i386 system, and
> out of the box jdk15 on FreeBSD performs extremely poorly.  The system
> is more than 90% idle, and profiling shows that the ~800 threads in the
> benchmark are spending most of their time doing short nanosleep() calls.
>
>
> I traced it to the following FreeBSD-specific hack in the jdk:
>
> // XXXBSD: understand meaning and workaround related to yield
> ...
> // XXXBSD: done differently in 1.3.1, take a look
> int os::sleep(Thread* thread, jlong millis, bool interruptible) {
>    assert(thread == Thread::current(),  "thread consistency check");
> ...
>
>    if (millis <= 0) {
>      // NOTE: workaround for bug 4338139
>      if (thread->is_Java_thread()) {
>        ThreadBlockInVM tbivm((JavaThread*) thread);
> // BSDXXX: Only use pthread_yield here and below if the system thread
> // scheduler gives time slices to lower priority threads when yielding.
> #ifdef __FreeBSD__
>        os_sleep(MinSleepInterval, interruptible);
> #else
>        pthread_yield();
> #endif
>
> When I removed this hack (i.e. revert to pthread_yield()) I got an
> immediate 7-fold performance increase, which brings FreeBSD performance
> on par with Solaris.
>
> What is the reason why this code is necessary?  Does FreeBSD's
> sched_yield() really have different semantics to the other operating
> systems, or was this a libkse bug that was being worked around?

Hello Kris,

I recall why I added the os_sleep() call. While working on the certification
testing one of the JCK tests was deadlocking. The test itself was faulty in
that it created a high priority thread in a tight yield loop. Since 
pthread_yield() on a single processor system will not yield to lower
priority threads, the higher priority thread effectively blocked the
lower priority thread from running and the test deadlocked.

I filed a formal appeal to have the JCK test corrected. However since the
api states:

   http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Thread.html#yield()
    "Causes the currently executing thread object to temporarily pause
     and allow other threads to execute." 

the appeal was denied. I further argued that many publications written
by or or-authored by Sun employees state that Thread.yield() can not
be relied upon in this fashion:

    "However, I believe there are other less authoritative sources
    of information that support the position that yield() can not
    expected to allow lower priority threads to execute in all cases.
    For example the following Sun document describes yield() as
    advisory only and not to depend on it for correctness:

   http://java.sun.com/j2se/1.5.0/docs/guide/vm/thread-priorities.html#general

    The document also refers to a Sun published book, "Effective
    Java Programming Language Guide" that describes yield()
    should not be relied on for correctness and portability.

    Another book I have specifically addresses the topic. It states
    that most schedulers do not stop the yielding thread from running
    in favor of a thread of lower priority. This is from the Sybex
    "Complete Java 2 Certification Study Guide" co-authored by
    a Sun employee.

    The publications I have referred to above present a case that the
    api description of yield() doesn't fully describe the expected behavior
    of the function when combined with thread priorities. While they are
    not the authoritive publications on the function, I think they represent
    the general historical interpretation of the expected behavior."

In the end I was not able to convince Sun to change the JCK test
so the os_sleep() call was added.

I see in my notes that kse on a single cpu system had the issue
but thr didn't (this is on 6.1-release). Perhaps now that thr is the
default on 7.0 this hack can be made conditional and only applied
to < 7.0.

The following are programs I wrote when I isolated the problem.
If the c program runs ok on 7.0 for both single cpu and mp then
remove the os_sleep() and try the java program. If that works too
then you're clear to make the os_sleep() hack only apply to <
7.0 and still be able to pass the certification tests.

Regards,
-Kurt

-----------
#include <pthread.h>
#include <unistd.h>
#include <signal.h>
#include <inttypes.h>
#include <time.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

volatile int init=0;
volatile int interrupt=0;

static void *
yielder(void *arg)
{
    init = 1;
    while (1) {
        pthread_yield();
    }
}

static void
sighandler(int sig)
{
    interrupt = 1;
    printf("sighandler\n");
static void
sighandler(int sig)
{
    interrupt = 1;
    printf("sighandler\n");
}

static void
waitForInit() {
    struct timespec t, rt;

static void
waitForInit() {
    struct timespec t, rt;

    while (init == 0) {
        t.tv_sec = 0;
        t.tv_nsec = 100000;
        nanosleep(&t, &rt);
    }
}

static void
waitForInterrupt() {
    struct timespec t, rt;

    while (interrupt == 0) {
        t.tv_sec = 0;
        t.tv_nsec = 100000;
        nanosleep(&t, &rt);
    }
}

int
main(int argc, char *argv[])
{
    pthread_t        yldr;
    pthread_attr_t   attr;
    struct sigaction act;

    /* Install a signal handler for SIGUSR1 */
    sigemptyset (&act.sa_mask);
    sigaddset (&act.sa_mask, SIGUSR1);
    act.sa_handler = sighandler;
    act.sa_flags = 0;
    sigaction (SIGUSR1, &act, NULL);

    pthread_attr_init(&attr);
    pthread_attr_setschedpolicy(&attr, SCHED_FIFO);

    pthread_create(&yldr, &attr, yielder, NULL);
    pthread_setprio(yldr, 16);
    waitForInit();
    if(pthread_kill(yldr, SIGUSR1) != 0)
        printf("pthread_kill failed with errno = %d\n", errno);
    waitForInterrupt();
}
---------------------
public class yieldTest {

    public static void main(String[] args) {
        yielderThread thread = new yielderThread();
        thread.setPriority(Thread.NORM_PRIORITY+1);
        thread.start();
        thread.waitForInit();
        thread.interrupt();
        try {
            thread.join();
        } catch (java.lang.InterruptedException ie) {
        }
    }
}

class yielderThread extends Thread {
    volatile boolean init = false;

    void waitForInit() {
        while (!init) {
            try {
                Thread.sleep(100);
            } catch (InterruptedException e) {
            }
        }
    }

    public void run() {
        init = true;
        while(!isInterrupted()) {
            yield();
        }
    }
}