High CPU usage in kernel on highly contended lock file

[Paul]

It seems that a lot of CPU resources are spend when trying to get exclusive lock on file from multiple processes concurrently. By multiple i mean hundreds.
It seems that there's an initial cost of fcntl() call. Each process that tries to lock the file consumes some amount of CPU and cools down.
However, each repeated fcntl() call will consume same amount of resources again. It seems as if entering the "wait queue" is expensive.

Environment:
#uname -a                        
FreeBSD test.com 11.1-STABLE FreeBSD 11.1-STABLE #0 r322650: Thu Aug 31 19:49:49 EEST 2017     root at test.com:/usr/obj/usr/src/sys/SERVER  amd64

Test case:

test.c:

#include <unistd.h>
#include <stdio.h>
#include <sys/fcntl.h>
#include <sys/time.h>
#include <signal.h>
#include <string.h>

static int child_count = 0;

static void schild_handler(int sig)
{
    --child_count;
}

static void alarm_handler(int sig)
{
}

void lock_write(int fd)
{
    struct flock fl;

    fl.l_type   = F_WRLCK;
    fl.l_whence = SEEK_SET;
    fl.l_start  = 0;
    fl.l_len    = 1;

    do {
        // Simulate interruption with alarm to re-enter the wait queue.
        alarm(1);
    }  while (fcntl(fd, F_SETLKW, &fl) < 0);
}

int main(int argc, char ** argv)
{
    if (argc < 2) {
        return 1;
    }

    signal(SIGCHLD, schild_handler);

    struct sigaction sig_action;

    memset(&sig_action, 0, sizeof sig_action);
    sig_action.sa_handler = alarm_handler;
    sigemptyset(&sig_action.sa_mask);
    sigaction(SIGALRM, &sig_action, NULL);

    int fd = open(argv[1], O_CREAT|O_RDWR, 0777);

    for (int i = 0; i < 300; ++i) {
        pid_t child_pid = fork();

        if (!child_pid) {
            // Lock the descriptor.
            lock_write(fd);

            // Simulate some work.
            sleep(1);
            return 0;
        }

        ++child_count;
    }

    do {
        printf("\rchild count: %5u\n", child_count);
        sleep(1);
    } while(child_count);

    return 0;
}

Commands:

# cd /tmp
# ~~~~~ Create test.c
# clang -o test test.c
# ./test 11111


Note that on linux, even if 1000 children are spawned instead of 300, none of them ever appear in the top.

This is a huge problem, because our current software uses lock files for sync purposes. And at times, when a lot of processes of said software are spawned (prime time), system becomes totally unresponsive with over 1000 LA.