RFC how to use kernel procs/threads efficiently

[Ian Lepore]

On Fri, 2017-10-06 at 19:02 +0000, Rick Macklem wrote:
> Hi,
> 
> I have now dropped the client side of Flexible File Layout for pNFS into head
> and I believe it is basically working.
> Currently when talking to mirrored DS servers, it does the Write and Commit
> RPCs to the mirrors serially. This works, but is inefficient w.r.t. elapsed to to
> completion.
> 
> To do them concurrently, I need separate kernel processes/threads to do them.
> I can think of two ways to do this:
> 1 - The code that I have running in projects/pnfs-planb-server for the pNFS server
>       side does a kproc_create() to create a kernel process that does the RPC and
>       then krpc_exit()s.
>       - This was easy to code and works. However, I am concerned that there is
>         going to be excessive overheads from doing all the kproc_create()s and
>         kproc_exit()s?
>        Anyone know if these calls will result in large overheads?
> 2 - I haven't coded this, but the other way I can think of to do this is to
>       create a pool of threads (kthread_create() is sufficient in this case, I
>       think?) and then hand each RPC to an available thread so it can do the RPC.
>       - Other than a little more complex coding, the main issue I see with this one
>         is "How many threads and when to create more/less of them.".
> 
> Anyhow, any comments w.r.t. the merits of either of the above approaches
> (or a suggestion of other ways to do this) would be appreciated, rick

taskqueue(9) is an existing mechanism to enqueue functions to execute
asynch using a pool of threads, but it doesn't answer the scalability
questions.  In fact it may make them harder, inasmuch as I don't think
there's a mechanism to dynamically adjust the number of threads after
first calling taskqueue_start_threads().

-- Ian