kqueue based thread pool

[David Xu]

Igor Sysoev wrote:
> The modern threads implementation works fine with program that requires
> fair threads scheduling, like Apache and MySQL.
> 
> However, there are programs (like varnish) that use FSM and thread pool.
> The threads allow to run on SMP and reduce blocking on disk I/O.
> These programs use an acceptor/listener thread and worker pool threads.
> In this model it's preferable to have one RUNnable worker thread per CPU and
> to start next one only when the current thread blocks. Also, when scheduler
> chooses next thread to run it should get the more recently run (cache hot)
> thread instead of fair scheduling. It's good to have some idle threads
> in pool, those run very seldom when active threads are blocked on I/O, etc.
> 
> Just wonder whether current scheduler is easy extendable to create thread
> pool controlled by kqueue: that is the kernel acts as acceptor/listener
> thread that wakes up worker threads waiting in kevent() ?
> 
> 

I have thought your idea for a while, if I understandd correctly,  I 
think you want FIFO
scheduling which does not do preemption. The problem in current system is
for 1:1 threading, doing realtime scheduling is not safe, kernel is not 
ready for
realtime scheduling, application code can starve other parts of system, 
this is why
SCHED_FIFO and SCHED_RR classes thread can only be created by root.
you may try libkse which doing FIFO in userland, just don't use SYSTEM 
scope thread,
it will work.
I am not sure kqueue will work with threading,  my brain tells me kqueue 
is not
multi-thread safe. ;-)

Regards,
David Xu