From nobody Sun Oct 13 21:38:29 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XRYdK5Ml5z5YZxt; Sun, 13 Oct 2024 21:38:29 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XRYdK4rfQz4k5k; Sun, 13 Oct 2024 21:38:29 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1728855509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Z3yO8UbSjCXjf4X/LYKSPIUYaqmDo/rzSfsiOnevxpM=; b=N5YSAJKOEtg2IGzIst8sojYpUcaR3D/n1QU2IHfmhkwDSV6LzBoIRs0EtCgaWv5vhnCyQX WTp5kc9+/+mO6R0xnl+L3L/6Y88zMCuMB1PjaCO4A4YGIAc1D72Gl4UIO1g9wRQyXt6Uhn CooNbTjxngzGUcZ37+JZRIxVa/HUYH1TenaJcuNn6A8KFuKhlNloDgand7vOLneBaup3MJ 5sc2bBLH85Vy5wYQkDrJcokjPLpl5L5+5+kX/8wNDDZN8lbRXfxbt+FgvPVLN3VKZTqbbB dMb1zHempEYZbaVf9UbgaBdLNLH/cD2EmfBiy1HLv/GkpY+/eb2zp+oUCPYbEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1728855509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Z3yO8UbSjCXjf4X/LYKSPIUYaqmDo/rzSfsiOnevxpM=; b=vCYlW1OejYvnQ5O4rCo4b4iZGtBrHIp75UZW/OtcWBqlZo9RLicejkf8H4E0Qq+0BYGPHm qZFI1b7ns3efypDWNYqGSg4k6vPfNl49itinvntq/QW9Cv8zNPYwdy6D1Vu+Iq/oiWuJbu XN8KjngpaLQiQaGdV/FJSxIEfzdc+q4BbjHMwF5E41EaQQzwYbuwzij71hSulgLS3yQOCn xEysOtMfb/C4KpZTUjlpm9sKkOMEAYF3fRiNe9e8Vjq1tfBkBJxZRrZulS4n80HLmpofPN qsa4gLPhrAVP6iVVBdqT7zmG3LvmRHkSLr7Ca9bT0fHgQ6av7eNl4NO2o8Kf1A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1728855509; a=rsa-sha256; cv=none; b=ha43pzik0ySUO2X5sZfZLbf/WUb4BK8QsovgCyJFA2BY1hN9ztM5LXANx6MlRHE6Qeyltb uQgwzGL+FjbluHdomRm81Mv8CDn+cZuBUsxuwxvtXtkuDfZz6BIdDMGILFBDlxnjf/y2Ls RHdkuhCLSMK4qUFIj5fnRYaaj+gaxBzyYIytJ0h4sgDyrbtZTC8wLLCTuFjW9xyQvyTJ4b qGGrKzgthUqu0ACX6GFJs3TW/fF/8VcrxZCTJdvyKjM8YQdSc2ybHxeqQVm0Cn1pAQJRyg lflPOTdWwYzSrq8yfbNml/xPjY58SGQ5adGhpiHtc5difZqs0wGSm7N78zaiRA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XRYdK4S07z191Z; Sun, 13 Oct 2024 21:38:29 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 49DLcTIk045507; Sun, 13 Oct 2024 21:38:29 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 49DLcTIG045504; Sun, 13 Oct 2024 21:38:29 GMT (envelope-from git) Date: Sun, 13 Oct 2024 21:38:29 GMT Message-Id: <202410132138.49DLcTIG045504@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Warner Losh Subject: git: c0e0e530ced0 - main - mps/mpr: Add workaround for firmware not responding to IOC_FACTS or IOC_INIT List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: imp X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: c0e0e530ced057502f51d7a6086857305e08fae0 Auto-Submitted: auto-generated The branch main has been updated by imp: URL: https://cgit.FreeBSD.org/src/commit/?id=c0e0e530ced057502f51d7a6086857305e08fae0 commit c0e0e530ced057502f51d7a6086857305e08fae0 Author: prateek sethi AuthorDate: 2024-10-13 18:38:54 +0000 Commit: Warner Losh CommitDate: 2024-10-13 21:38:01 +0000 mps/mpr: Add workaround for firmware not responding to IOC_FACTS or IOC_INIT Sometimes, especially with older firmware, mps(4) would have trouble initializing the card in one of these two steps. Add in a retry after a short delay. Sean Bruno and Stephen McConnell thought this was OK in the bug discussions, but never committed it. Steve indicated the delay might not be necessary, but the OP clearly needed to make it longer to make things work. I've kept the delay, and added the suggested comment. Ported the iocfacts part to mpr as well, since we see similar errors about once every month or two over a few thousand controllers at work. We've not seen it with IOC_INIT as far back as I can query the error log database, so I didn't port that forward. We'll see if this helps, but won't know for sure until next year (so I'm committing it now since it won't hurt and might help). We usually see this failure in connection with complicated recovery operations with a drive that's failing, though, at least in the last year's worth of failures. It's not clear this is the same as OP or not. PR: 212841 Sponsored by: Netflix Co-authored-by: imp --- sys/dev/mpr/mpr.c | 23 ++++++++++++++++++----- sys/dev/mps/mps.c | 38 ++++++++++++++++++++++++++++++-------- 2 files changed, 48 insertions(+), 13 deletions(-) diff --git a/sys/dev/mpr/mpr.c b/sys/dev/mpr/mpr.c index d5c02f9608ca..d1c572e40669 100644 --- a/sys/dev/mpr/mpr.c +++ b/sys/dev/mpr/mpr.c @@ -1195,7 +1195,7 @@ mpr_get_iocfacts(struct mpr_softc *sc, MPI2_IOC_FACTS_REPLY *facts) { MPI2_DEFAULT_REPLY *reply; MPI2_IOC_FACTS_REQUEST request; - int error, req_sz, reply_sz; + int error, req_sz, reply_sz, retry = 0; MPR_FUNCTRACE(sc); mpr_dprint(sc, MPR_INIT, "%s entered\n", __func__); @@ -1204,13 +1204,26 @@ mpr_get_iocfacts(struct mpr_softc *sc, MPI2_IOC_FACTS_REPLY *facts) reply_sz = sizeof(MPI2_IOC_FACTS_REPLY); reply = (MPI2_DEFAULT_REPLY *)facts; + /* + * Retry sending the initialization sequence. Sometimes, especially with + * older firmware, the initialization process fails. Retrying allows the + * error to clear in the firmware. + */ bzero(&request, req_sz); request.Function = MPI2_FUNCTION_IOC_FACTS; - error = mpr_request_sync(sc, &request, reply, req_sz, reply_sz, 5); - - adjust_iocfacts_endianness(facts); - mpr_dprint(sc, MPR_TRACE, "facts->IOCCapabilities 0x%x\n", facts->IOCCapabilities); + while (retry < 5) { + error = mpr_request_sync(sc, &request, reply, req_sz, reply_sz, 5); + if (error == 0) + break; + mpr_dprint(sc, MPR_FAULT, "%s failed retry %d\n", __func__, retry); + DELAY(1000); + retry++; + } + if (error == 0) { + adjust_iocfacts_endianness(facts); + mpr_dprint(sc, MPR_TRACE, "facts->IOCCapabilities 0x%x\n", facts->IOCCapabilities); + } mpr_dprint(sc, MPR_INIT, "%s exit, error= %d\n", __func__, error); return (error); } diff --git a/sys/dev/mps/mps.c b/sys/dev/mps/mps.c index adad2450a3d4..9dfa0471ac0a 100644 --- a/sys/dev/mps/mps.c +++ b/sys/dev/mps/mps.c @@ -1127,7 +1127,7 @@ mps_get_iocfacts(struct mps_softc *sc, MPI2_IOC_FACTS_REPLY *facts) { MPI2_DEFAULT_REPLY *reply; MPI2_IOC_FACTS_REQUEST request; - int error, req_sz, reply_sz; + int error, req_sz, reply_sz, retry = 0; MPS_FUNCTRACE(sc); mps_dprint(sc, MPS_INIT, "%s entered\n", __func__); @@ -1136,10 +1136,21 @@ mps_get_iocfacts(struct mps_softc *sc, MPI2_IOC_FACTS_REPLY *facts) reply_sz = sizeof(MPI2_IOC_FACTS_REPLY); reply = (MPI2_DEFAULT_REPLY *)facts; + /* + * Retry sending the initialization sequence. Sometimes, especially with + * older firmware, the initialization process fails. Retrying allows the + * error to clear in the firmware. + */ bzero(&request, req_sz); request.Function = MPI2_FUNCTION_IOC_FACTS; - error = mps_request_sync(sc, &request, reply, req_sz, reply_sz, 5); - mps_dprint(sc, MPS_INIT, "%s exit error= %d\n", __func__, error); + while (retry < 5) { + error = mps_request_sync(sc, &request, reply, req_sz, reply_sz, 5); + if (error == 0) + break; + mps_dprint(sc, MPS_FAULT, "%s failed retry %d\n", __func__, retry); + DELAY(1000); + retry++; + } return (error); } @@ -1149,7 +1160,7 @@ mps_send_iocinit(struct mps_softc *sc) { MPI2_IOC_INIT_REQUEST init; MPI2_DEFAULT_REPLY reply; - int req_sz, reply_sz, error; + int req_sz, reply_sz, error, retry = 0; struct timeval now; uint64_t time_in_msec; @@ -1193,10 +1204,21 @@ mps_send_iocinit(struct mps_softc *sc) time_in_msec = (now.tv_sec * 1000 + now.tv_usec/1000); init.TimeStamp.High = htole32((time_in_msec >> 32) & 0xFFFFFFFF); init.TimeStamp.Low = htole32(time_in_msec & 0xFFFFFFFF); - - error = mps_request_sync(sc, &init, &reply, req_sz, reply_sz, 5); - if ((reply.IOCStatus & MPI2_IOCSTATUS_MASK) != MPI2_IOCSTATUS_SUCCESS) - error = ENXIO; + /* + * Retry sending the initialization sequence. Sometimes, especially with + * older firmware, the initialization process fails. Retrying allows the + * error to clear in the firmware. + */ + while (retry < 5) { + error = mps_request_sync(sc, &init, &reply, req_sz, reply_sz, 5); + if ((reply.IOCStatus & MPI2_IOCSTATUS_MASK) != MPI2_IOCSTATUS_SUCCESS) + error = ENXIO; + if (error == 0) + break; + mps_dprint(sc, MPS_FAULT, "%s failed retry %d\n", __func__, retry); + DELAY(1000); + retry++; + } mps_dprint(sc, MPS_INIT, "IOCInit status= 0x%x\n", reply.IOCStatus); mps_dprint(sc, MPS_INIT, "%s exit\n", __func__);