irq cpu binding

[Slawa Olhovchenkov]

On Tue, Mar 31, 2015 at 04:14:04PM -0700, Adrian Chadd wrote:

> You also have to do the taskqueue threads.

# procstat -ta | grep ix
    0 101305 kernel           ix0 que            6    8 sleep   -         
    0 101306 kernel           ix0 que            7    8 sleep   -         
    0 101307 kernel           ix0 que            8    8 sleep   -         
    0 101308 kernel           ix0 linkq          7    8 sleep   -         
    0 101309 kernel           ix1 que            9    8 sleep   -         
    0 101310 kernel           ix1 que           10    8 sleep   -         
    0 101311 kernel           ix1 que           11    8 sleep   -         
    0 101312 kernel           ix1 linkq          1    8 sleep   -         
   12 100064 intr             irq270: ix0:que    6    8 run     -         
   12 100066 intr             irq271: ix0:que    7    8 run     -         
   12 100068 intr             irq272: ix0:que    8    8 run     -         
   12 100070 intr             irq273: ix0:link   5    8 wait    -         
   12 100072 intr             irq274: ix1:que    9    8 wait    -         
   12 100074 intr             irq275: ix1:que   10    8 wait    -         
   12 100076 intr             irq276: ix1:que   11    8 run     -         
   12 100078 intr             irq277: ix1:link   9    8 wait    -         
# cpuset -g -t 101305
tid 101305 mask: 6
# cpuset -g -t 101306
tid 101306 mask: 7
# cpuset -g -t 101307
tid 101307 mask: 8
# cpuset -g -t 101309
tid 101309 mask: 9
# cpuset -g -t 101310
tid 101310 mask: 10
# cpuset -g -t 101311
tid 101311 mask: 11
# cpuset -g -t 100064
tid 100064 mask: 6
# cpuset -g -t 100066
tid 100066 mask: 7
# cpuset -g -t 100068
tid 100068 mask: 8
# cpuset -g -t 100072
tid 100072 mask: 9
# cpuset -g -t 100074
tid 100074 mask: 10
# cpuset -g -t 100076
tid 100076 mask: 11

Or you talk abot somewere different?

> (I mean, pmc could also be broken...)

Why?

> -a
> 
> 
> On 31 March 2015 at 15:59, Slawa Olhovchenkov <slw at zxy.spb.ru> wrote:
> > On Sat, Mar 28, 2015 at 12:33:52PM -0700, Adrian Chadd wrote:
> >
> >> That's done deferred by the bus interrupt wiring. That's something
> >> John's been looking into as part of the general NUMA work (and I'm
> >> trying to debug right now, on dual-socket boxes with ixgbe. :-)
> >>
> >> Look at bus_bind_intr() and the twisty path to intr_event_bind(), then
> >> x86/x86/intr_machdep.c:intr_assign_cpu(), then intr_shuffle_cpus() at
> >> boot, versus what happens via calls to pic_assign_cpu to setup the
> >> wiring.
> >
> > I am do simple, ugle hack ixgbe driver for let start cpu binding.
> > I am still see ixgbe in pmc output.
> >
> > What may be wrong?
> > What may be miss?
> >
> > =====
> > static int ixgbe_start_cpu = 0;
> > TUNABLE_INT("hw.ix.start_cpu", &ixgbe_start_cpu);
> > SYSCTL_INT(_hw_ix, OID_AUTO, start_cpu, CTLFLAG_RDTUN, &ixgbe_start_cpu, 0,
> >     "Start CPU for next IRQ binding");
> >
> > [...]
> >                 if (adapter->num_queues > 1)
> >                         bus_bind_intr(dev, que->res, i+ixgbe_start_cpu);
> >
> > #ifndef IXGBE_LEGACY_TX
> >                 TASK_INIT(&txr->txq_task, 0, ixgbe_deferred_mq_start, txr);
> > #endif
> >                 TASK_INIT(&que->que_task, 0, ixgbe_handle_que, que);
> >                 que->tq = taskqueue_create_fast("ixgbe_que", M_NOWAIT,
> >                     taskqueue_thread_enqueue, &que->tq);
> >                 taskqueue_start_threads(&que->tq, 1, PI_NET, "%s que",
> >                     device_get_nameunit(adapter->dev));
> >         }
> >         ixgbe_start_cpu += adapter->num_queues;
> >