git: 2f32a971b7f9 - main - Wait longer for a previous IPI to be sent

[Eric van Gyzen]

The branch main has been updated by vangyzen:

URL: https://cgit.FreeBSD.org/src/commit/?id=2f32a971b7f936170b4d61318238e6cf89f198b5

commit 2f32a971b7f936170b4d61318238e6cf89f198b5
Author:     Eric van Gyzen <vangyzen at FreeBSD.org>
AuthorDate: 2021-04-26 15:01:17 +0000
Commit:     Eric van Gyzen <vangyzen at FreeBSD.org>
CommitDate: 2021-04-30 18:32:29 +0000

    Wait longer for a previous IPI to be sent
    
    When sending an IPI, if a previous IPI is still pending delivery,
    native_lapic_ipi_vectored() waits for the previous IPI to be sent.
    We've seen a few inexplicable panics with the current timeout of 50 ms.
    Increase the timeout to 1 second and make it tunable.
    
    No hardware specification mentions a timeout in this case; I checked
    the Intel SDM, Intel MP spec, and Intel x2APIC spec.  Linux and illumos
    wait forever.  In Linux, see __default_send_IPI_shortcut() in
    arch/x86/kernel/apic/ipi.c.  In illumos, see apic_send_ipi() in
    usr/src/uts/i86pc/io/pcplusmp/apic_common.c.  However, misbehaving hardware
    could hang the system if we wait forever.
    
    Reviewed by:    mav kib
    MFC after:      1 week
    Sponsored by:   Dell EMC Isilon
    Differential Revision: https://reviews.freebsd.org/D29942
---
 sys/x86/x86/local_apic.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c
index 85c3cfb69277..36010fe38102 100644
--- a/sys/x86/x86/local_apic.c
+++ b/sys/x86/x86/local_apic.c
@@ -202,6 +202,7 @@ static u_long lapic_timer_divisor, count_freq;
 static struct eventtimer lapic_et;
 #ifdef SMP
 static uint64_t lapic_ipi_wait_mult;
+static int __read_mostly lapic_ds_idle_timeout = 1000000;
 #endif
 unsigned int max_apic_id;
 
@@ -212,6 +213,11 @@ SYSCTL_INT(_hw_apic, OID_AUTO, eoi_suppression, CTLFLAG_RD,
     &lapic_eoi_suppression, 0, "");
 SYSCTL_INT(_hw_apic, OID_AUTO, timer_tsc_deadline, CTLFLAG_RD,
     &lapic_timer_tsc_deadline, 0, "");
+#ifdef SMP
+SYSCTL_INT(_hw_apic, OID_AUTO, ds_idle_timeout, CTLFLAG_RWTUN,
+    &lapic_ds_idle_timeout, 0,
+    "timeout (in us) for APIC Delivery Status to become Idle (xAPIC only)");
+#endif
 
 static void lapic_calibrate_initcount(struct lapic *la);
 static void lapic_calibrate_deadline(struct lapic *la);
@@ -2035,7 +2041,6 @@ native_lapic_ipi_raw(register_t icrlo, u_int dest)
 	}
 }
 
-#define	BEFORE_SPIN	50000
 #ifdef DETECT_DEADLOCK
 #define	AFTER_SPIN	50
 #endif
@@ -2082,7 +2087,7 @@ native_lapic_ipi_vectored(u_int vector, int dest)
 	icrlo |= APIC_DESTMODE_PHY | APIC_TRIGMOD_EDGE | APIC_LEVEL_ASSERT;
 
 	/* Wait for an earlier IPI to finish. */
-	if (!lapic_ipi_wait(BEFORE_SPIN)) {
+	if (!lapic_ipi_wait(lapic_ds_idle_timeout)) {
 		if (KERNEL_PANICKED())
 			return;
 		else