svn commit: r279699 - in head/sys: amd64/amd64 i386/i386

[John Baldwin]

On Monday, March 09, 2015 07:42:52 PM Konstantin Belousov wrote:
> On Fri, Mar 06, 2015 at 08:34:28PM +0000, John Baldwin wrote:
> > Author: jhb
> > Date: Fri Mar  6 20:34:28 2015
> > New Revision: 279699
> > URL: https://svnweb.freebsd.org/changeset/base/279699
> > 
> > Log:
> >   Only schedule interrupts on a single hyperthread of a modern Intel CPU
> >   core
> >   by default.  Previously we used a single hyperthread on Pentium4-era
> >   cores but used both hyperthreads on more recent CPUs.
> >   
> >   MFC after:	2 weeks
> > 
> > Modified:
> >   head/sys/amd64/amd64/mp_machdep.c
> >   head/sys/i386/i386/mp_machdep.c
> > 
> > Modified: head/sys/amd64/amd64/mp_machdep.c
> > ==========================================================================
> > ==== --- head/sys/amd64/amd64/mp_machdep.c	Fri Mar  6 16:43:54
> > 2015	(r279698) +++ head/sys/amd64/amd64/mp_machdep.c	Fri Mar  6 20:34:28
> > 2015	(r279699) @@ -828,8 +828,8 @@ set_interrupt_apic_ids(void)
> > 
> >  			continue;
> >  		
> >  		/* Don't let hyperthreads service interrupts. */
> > 
> > -		if (hyperthreading_cpus > 1 &&
> > -		    apic_id % hyperthreading_cpus != 0)
> > +		if (cpu_logical > 1 &&
> > +		    apic_id % cpu_logical != 0)
> > 
> >  			continue;
> >  		
> >  		intr_add_cpu(i);
> > 
> > Modified: head/sys/i386/i386/mp_machdep.c
> > ==========================================================================
> > ==== --- head/sys/i386/i386/mp_machdep.c	Fri Mar  6 16:43:54
> > 2015	(r279698) +++ head/sys/i386/i386/mp_machdep.c	Fri Mar  6 20:34:28
> > 2015	(r279699) @@ -842,8 +842,8 @@ set_interrupt_apic_ids(void)
> > 
> >  			continue;
> >  		
> >  		/* Don't let hyperthreads service interrupts. */
> > 
> > -		if (hyperthreading_cpus > 1 &&
> > -		    apic_id % hyperthreading_cpus != 0)
> > +		if (cpu_logical > 1 &&
> > +		    apic_id % cpu_logical != 0)
> > 
> >  			continue;
> >  		
> >  		intr_add_cpu(i);
> 
> BTW, this sounds somewhat backward from the intention of the HTT/SMT.
> The feature was aimed to reduce latency of memory or device registers
> reads by allowing several contexts to stuck on the cache line fill
> or waiting for the device response to read request.
> 
> Since typical interrupt handler just EOIs the source or does nothing at
> all to the source, what is the reasoning behind the change ?

It also affects where ithreads are run.  SCHED_ULE does a soft binding of
ithreads to the CPU that scheduled them (i.e. the CPU that received the
interrupt).  It also matters for bus_get_cpus(9) as this will affect the set 
of CPUs that returns so that by default multi queue NICs will only schedule
one interrupt + bound ithread (and in some cases bound taskqueue thread) per-
core rather than one per-thread.

I have thought about having a separate tunable to control whether or not this 
is in effect if there are any workloads where having interrupts on all threads 
in a core helps (for all the cases I am aware of, people end up disabling HTT 
in the BIOS to get this effect instead).  Ultimately I'd like to make it 
possible to manipulate the set of CPUs used for interrupts directly, but 
that's a larger change.

-- 
John Baldwin