panic: ipi_send: couldn't send ipi

[Ken Smith]

On Sat, Sep 25, 2004 at 12:07:41AM -0700, Kris Kennaway wrote:
> Another panic on SMP sparc:
> 
> panic: ipi_send: couldn't send ipi
> cpuid = 1
> KDB: enter: panic
> [thread 100158]
> Stopped at      kdb_enter+0x38: ta              %xcc, 1
> db> trace
> panic() at panic+0x19c
> cpu_ipi_send() at cpu_ipi_send+0xb8
> cpu_ipi_selected() at cpu_ipi_selected+0x38
> spitfire_icache_page_inval() at spitfire_icache_page_inval+0x70
> pmap_enter() at pmap_enter+0x27c
> vm_fault() at vm_fault+0x1200
> trap_pfault() at trap_pfault+0x1e4
> trap() at trap+0x208
> -- fast data access mmu miss tar=0x4041e797 %o7=0x402c2f48 --
> userland() at 0x402c2f5c
> user trace: trap %o7=0x402c2f48
> 

Would someone(s) who have been having this problem mind trying
the following patch?  I'm a bit nervous about just "blindly"
raising the IPI_RETRIES constant to something huge, I would
like to try and track this down a bit to find out a bit more
about what's going on.

This patch is a bit weird, but it should report on the console
when it finds a need to bump the number of retries (and it's
starting off with a higher default number than before - the
PR that someone cited reported test numbers in the 2000 range
with that test patch).  If you find that it stops the panic's
I'm interested in finding what it winds up bumping the loop
counter to, and if anyone knows what sort of activity triggers
the delays in the ipi delivery I'm interested...

I've compile-tested this but I'm afraid I sorta loaned my test
MP machines to someone for a while so I can't try to exercise
this here at the moment.  :-(

Thanks...

Index: sys/sparc64/include/smp.h
===================================================================
RCS file: /home/ncvs/src/sys/sparc64/include/smp.h,v
retrieving revision 1.16
diff -u -r1.16 smp.h
--- sys/sparc64/include/smp.h	8 Apr 2003 06:35:08 -0000	1.16
+++ sys/sparc64/include/smp.h	27 Sep 2004 17:01:52 -0000
@@ -45,7 +45,9 @@
 #define	IPI_RENDEZVOUS	PIL_RENDEZVOUS
 #define	IPI_STOP	PIL_STOP
 
-#define	IPI_RETRIES	100
+#define	IPI_RETRIES_START	1000
+#define	IPI_RETRIES_INCREMENT	1000
+#define	IPI_RETRIES_MAX		10000
 
 struct cpu_start_args {
 	u_int	csa_count;
Index: sys/sparc64/sparc64/mp_machdep.c
===================================================================
RCS file: /home/ncvs/src/sys/sparc64/sparc64/mp_machdep.c,v
retrieving revision 1.27
diff -u -r1.27 mp_machdep.c
--- sys/sparc64/sparc64/mp_machdep.c	27 Sep 2004 16:06:38 -0000	1.27
+++ sys/sparc64/sparc64/mp_machdep.c	27 Sep 2004 19:13:00 -0000
@@ -107,6 +107,8 @@
 
 static volatile u_int	shutdown_cpus;
 
+int	ipi_retries_max = IPI_RETRIES_START;
+
 void cpu_mp_unleash(void *);
 SYSINIT(cpu_mp_unleash, SI_SUB_SMP, SI_ORDER_FIRST, cpu_mp_unleash, NULL);
 
@@ -429,7 +431,7 @@
 
 	KASSERT((ldxa(0, ASI_INTR_DISPATCH_STATUS) & IDR_BUSY) == 0,
 	    ("cpu_ipi_send: outstanding dispatch"));
-	for (i = 0; i < IPI_RETRIES; i++) {
+	for (i = 0; i <= ipi_retries_max; i++) {
 		s = intr_disable();
 		stxa(AA_SDB_INTR_D0, ASI_SDB_INTR_W, d0);
 		stxa(AA_SDB_INTR_D1, ASI_SDB_INTR_W, d1);
@@ -441,6 +443,11 @@
 		intr_restore(s);
 		if ((ldxa(0, ASI_INTR_DISPATCH_STATUS) & IDR_NACK) == 0)
 			return;
+		if (i == ipi_retries_max && ipi_retries_max < IPI_RETRIES_MAX) {
+			ipi_retries_max += IPI_RETRIES_INCREMENT;
+			printf("cpu_ipi_send: raised ipi_retries_max to %d\n",
+			    ipi_retries_max);
+		}
 	}
 	if (
 #ifdef KDB


-- 
						Ken Smith
- From there to here, from here to      |       kensmith at cse.buffalo.edu
  there, funny things are everywhere.   |
                      - Theodore Geisel |