svn commit: r242014 - head/sys/kern

[Jim Harris]

Author: jimharris
Date: Wed Oct 24 18:36:41 2012
New Revision: 242014
URL: http://svn.freebsd.org/changeset/base/242014

Log:
  Pad tdq_lock to avoid false sharing with tdq_load and tdq_cpu_idle.
  
  This enables CPU searches (which read tdq_load) to operate independently
  of any contention on the spinlock.  Some scheduler-intensive workloads
  running on an 8C single-socket SNB Xeon show considerable improvement with
  this change (2-3% perf improvement, 5-6% decrease in CPU util).
  
  Sponsored by:	Intel
  Reviewed by:	jeff

Modified:
  head/sys/kern/sched_ule.c

Modified: head/sys/kern/sched_ule.c
==============================================================================
--- head/sys/kern/sched_ule.c	Wed Oct 24 18:33:44 2012	(r242013)
+++ head/sys/kern/sched_ule.c	Wed Oct 24 18:36:41 2012	(r242014)
@@ -223,8 +223,13 @@ static int sched_idlespinthresh = -1;
  * locking in sched_pickcpu();
  */
 struct tdq {
-	/* Ordered to improve efficiency of cpu_search() and switch(). */
+	/* 
+	 * Ordered to improve efficiency of cpu_search() and switch().
+	 * tdq_lock is padded to avoid false sharing with tdq_load and
+	 * tdq_cpu_idle.
+	 */
 	struct mtx	tdq_lock;		/* run queue lock. */
+	char		pad[64 - sizeof(struct mtx)];
 	struct cpu_group *tdq_cg;		/* Pointer to cpu topology. */
 	volatile int	tdq_load;		/* Aggregate load. */
 	volatile int	tdq_cpu_idle;		/* cpu_idle() is active. */