RFC how to use kernel procs/threads efficiently

[Rick Macklem]

Ian Lepore wrote:
>On Fri, 2017-10-06 at 19:02 +0000, Rick Macklem wrote:
>> Hi,
>>
>> I have now dropped the client side of Flexible File Layout for pNFS into head
>> and I believe it is basically working.
>> Currently when talking to mirrored DS servers, it does the Write and Commit
>> RPCs to the mirrors serially. This works, but is inefficient w.r.t. elapsed to to
>> completion.
>>
>> To do them concurrently, I need separate kernel processes/threads to do them.
>> I can think of two ways to do this:
>> 1 - The code that I have running in projects/pnfs-planb-server for the pNFS server
>>       side does a kproc_create() to create a kernel process that does the RPC and
>>       then krpc_exit()s.
>>       - This was easy to code and works. However, I am concerned that there is
>>         going to be excessive overheads from doing all the kproc_create()s and
>>         kproc_exit()s?
>>        Anyone know if these calls will result in large overheads?
>> 2 - I haven't coded this, but the other way I can think of to do this is to
>>       create a pool of threads (kthread_create() is sufficient in this case, I
>>       think?) and then hand each RPC to an available thread so it can do the RPC.
>>       - Other than a little more complex coding, the main issue I see with this one
>>         is "How many threads and when to create more/less of them.".
>>
>> Anyhow, any comments w.r.t. the merits of either of the above approaches
>> (or a suggestion of other ways to do this) would be appreciated, rick
>
>taskqueue(9) is an existing mechanism to enqueue functions to execute
>asynch using a pool of threads, but it doesn't answer the scalability
>questions.  In fact it may make them harder, inasmuch as I don't think
>there's a mechanism to dynamically adjust the number of threads after
>first calling taskqueue_start_threads().
Hmm, yes. Thanks for the pointer. I hadn't read "man taskqueue" until now.
The kernel RPC doesn't use this and I suspect that it is because of what you
said w.r.t. dynamically adjusting the # of threads.
However, it does save "hand coding" the queues for #2 and I'm lazy (plus
don't believe reinventing the wheel is the best plan).

I think I will try using taskqueue and just have a sysctl for #of-threads.
(Actually most of the code ends up the same, because basically they all
 end up with a function with a single argument that does the RPC. The
 only difference is what call starts the RPC.)

Anyone else have comments? rick