svn commit: r304313 - head/sys/net

[Robert N. M. Watson]

On 20 Aug 2016, at 21:27, Adrian Chadd <adrian at freebsd.org> wrote:

> I wonder if the right-er thing to do here is to allow the cpuid to be
> whatever it needs to be, but limit the cpuid lookups when it resolves
> to a netisr array.
> 
> that'd mean the hybrid model would still return the current CPU up to
> the max CPU id, but anything trying to queue into a netisr would not
> overflow the netisr queue count.


I think some refinement of model is required. The original intent was that “workstreams” represented ordered sets of queues being delivered for deferred dispatch across the set of CPUs (hence “nws” being allocated using DPCPU). When performing direct dispatch, as a matter of convenience, we also bill work performed on a CPU to its workstream:

 1109         /*
 1110          * If direct dispatch is forced, then unconditionally dispatch
 1111          * without a formal CPU selection.  Borrow the current CPU's stats,
 1112          * even if there's no worker on it.  In this case we don't update
 1113          * nws_flags because all netisr processing will be source ordered due
 1114          * to always being forced to directly dispatch.
 1115          */

In hybrid mode, the world is a little different, because we are willing to direct dispatch hybrid work *only* if the worker thread isn’t already processing the workstream:

 1147         /*-
 1148          * We are willing to direct dispatch only if three conditions hold:
 1149          *
 1150          * (1) The netisr worker isn't already running,
 1151          * (2) Another thread isn't already directly dispatching, and
 1152          * (3) The netisr hasn't already been woken up.
 1153          */

This is important because if the workstream already has a backlog of work, but the thread isn’t yet running, we must enqueue the work to ensure it remains ordered with respect to other work in the workstream.

Assuming we wish to retain the current workstream model, then we need to adapt the code a bit to handle the case where we still have one workstream per CPU, but where we are only willing to perform deferred dispatch (or hybrid dispatch) to a CPU that has a worker thread. We would then bill to any CPU’s workstream for true direct dispatch, but for hybrid dispatch, continue to always bill a CPU that has a worker thread to which we were willing to assign the work.

I.e., as I think you’re suggesting, we probably need to tweak the functions slightly to differentiate between “you can run it here and must use curcpu’s workstream” and “you can run it here but must use a specific CPU’s workstream” — the former being true direct dispatch, and the latter being hybrid dispatch where the netisr in question isn’t already running, but we bill it to that workstream/netisr even though we run it on the current CPU.

Does this make sense?

Robert