From nobody Mon Oct 21 18:39:35 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XXPHC48v1z5Zcws; Mon, 21 Oct 2024 18:39:35 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XXPHC2m0qz4GdP; Mon, 21 Oct 2024 18:39:35 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1729535975; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=4lYi8y1pcJlpPtMs9ZXvznZYDr+mpmSfx+2MZOFSrXA=; b=ijpZ9VBJ5rGNFjfWoJ94Mjumqa76lQkRTZz/Ezh6ObW6rtqm2ZS5MTfEsgwtKkIFBYpQMK LfsemJr6N8LIiOILwxjpIqYvUMJrGAs7sw9fhwwiyyYkT2FIu3QiosmIhL6sv34u0JveEl qsUicxygRGOYLWV5qKhC0vZALXw22XZ5R6WoUqpZZkZAayT6hKACHUGG0Hd6NtC4YL7oPm 8fovtVly8tFNXs9uXi2XMIgM/y9YOsC067LiDWi9xnxatjB2jW++bohW6kV4UFBGEZ5JZh GtiLVQvFdblBLhiTx+Np0dM9z94iUtNqrYf6xCd1IaBO8f5/zPyGokZlQ2W8qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1729535975; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=4lYi8y1pcJlpPtMs9ZXvznZYDr+mpmSfx+2MZOFSrXA=; b=EGK/an5uMeRBw34rVpNTcjMm6sguplpvZHYatf40ih4YA8dtMB4zAM6vHjNdUoeuuig9WY PbOVhaBi8A7TT1vEmcB7Cev1lRFmVE7BG0KRnBcNm4xI2FTrmqc1gYEbYYB6eP4LzTQJkX DcjbvB/fa0tKRY08kygdDoGndkWXf8p4Pw/wBGSJRyw8rQiPoKbkya7vJLaL1ilFZenD3y zV4Va2cnhVM+vrtq4YjSZvYZAbXPssLW4dijxaMMhHvsPBBZhDduFptgOBG7oV2H5eX4UC owo0XMIH7WSMXhtUDpm289rzVTdykvJFiSPtbimlaWP8EWHju/PNBz6avB4ZPg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1729535975; a=rsa-sha256; cv=none; b=ovp8dCVcvheEUH9bxDqQ96R5bOb9lEVzByyVQUhTbMjenRJT+TRHrQqNoecQGB/JbsnxP3 YeEdx+zSXKJIBFaVTN7vrFCWHgM5tRInSl1har7Eg/X/5PUyOImQAHKpCRx1utcClbT4So mTvnh5f/DB2T/9DJdaQs7+NZ/h+3oq9vhbc9a8ymy8W9ZlG4P4XJGcivieQq8vvp1CUP5m DQFSXbOLL6i2sWLqKW+7+6WxNF5WRay1jQDIjDIjR5siCsDDnn51Dj51rCuYNh6k2nmfqx R0BEVy/WgNzx6kLZ36IJHAz7ppRSqE8qLk509t19D4d3X/l2pbFy683P7fJo0w== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XXPHC279SzhVm; Mon, 21 Oct 2024 18:39:35 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 49LIdZtZ037403; Mon, 21 Oct 2024 18:39:35 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 49LIdZQN037400; Mon, 21 Oct 2024 18:39:35 GMT (envelope-from git) Date: Mon, 21 Oct 2024 18:39:35 GMT Message-Id: <202410211839.49LIdZQN037400@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Navdeep Parhar Subject: git: 0eea8754aefc - stable/14 - cxgbe(4): Allow t4_tom to be unloaded safely. List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: np X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 0eea8754aefc64ce862705a13215837f50ee8983 Auto-Submitted: auto-generated The branch stable/14 has been updated by np: URL: https://cgit.FreeBSD.org/src/commit/?id=0eea8754aefc64ce862705a13215837f50ee8983 commit 0eea8754aefc64ce862705a13215837f50ee8983 Author: Navdeep Parhar AuthorDate: 2024-09-29 23:29:41 +0000 Commit: Navdeep Parhar CommitDate: 2024-10-21 17:15:11 +0000 cxgbe(4): Allow t4_tom to be unloaded safely. * Disable IFCAP_TOE automatically on all ifnets on all adapters during unload. This is user-friendly and avoids panics due to stale ifnet state after t4_tom is unloaded. * Do not allow unload if tids are in use by the TOE on any adapter. Reported by: Bimal Abraham @ Chelsio Sponsored by: Chelsio Communications (cherry picked from commit 9ba8670a8b175de79ea087688f51595b4f2db862) --- sys/dev/cxgbe/adapter.h | 1 + sys/dev/cxgbe/t4_main.c | 22 +++++++++++----------- sys/dev/cxgbe/tom/t4_tom.c | 31 +++++++++++++++++++++++-------- 3 files changed, 35 insertions(+), 19 deletions(-) diff --git a/sys/dev/cxgbe/adapter.h b/sys/dev/cxgbe/adapter.h index 2769277e8411..862a90e8a441 100644 --- a/sys/dev/cxgbe/adapter.h +++ b/sys/dev/cxgbe/adapter.h @@ -1394,6 +1394,7 @@ void cxgbe_media_status(if_t, struct ifmediareq *); void t4_os_cim_err(struct adapter *); int suspend_adapter(struct adapter *); int resume_adapter(struct adapter *); +int toe_capability(struct vi_info *, bool); #ifdef KERN_TLS /* t6_kern_tls.c */ diff --git a/sys/dev/cxgbe/t4_main.c b/sys/dev/cxgbe/t4_main.c index e33e732a7fa2..f4e6b41d9fb4 100644 --- a/sys/dev/cxgbe/t4_main.c +++ b/sys/dev/cxgbe/t4_main.c @@ -869,7 +869,6 @@ static int stop_lld(struct adapter *); static inline int restart_adapter(struct adapter *); static int restart_lld(struct adapter *); #ifdef TCP_OFFLOAD -static int toe_capability(struct vi_info *, bool); static int deactivate_all_uld(struct adapter *); static void stop_all_uld(struct adapter *); static void restart_all_uld(struct adapter *); @@ -12501,7 +12500,7 @@ t4_ioctl(struct cdev *dev, unsigned long cmd, caddr_t data, int fflag, } #ifdef TCP_OFFLOAD -static int +int toe_capability(struct vi_info *vi, bool enable) { int rc; @@ -12567,6 +12566,7 @@ toe_capability(struct vi_info *vi, bool enable) if (isset(&sc->offload_map, pi->port_id)) { /* TOE is enabled on another VI of this port. */ + MPASS(pi->uld_vis > 0); pi->uld_vis++; return (0); } @@ -12592,17 +12592,17 @@ toe_capability(struct vi_info *vi, bool enable) if (!uld_active(sc, ULD_ISCSI)) (void) t4_activate_uld(sc, ULD_ISCSI); - pi->uld_vis++; - setbit(&sc->offload_map, pi->port_id); + if (pi->uld_vis++ == 0) + setbit(&sc->offload_map, pi->port_id); } else { - pi->uld_vis--; - - if (!isset(&sc->offload_map, pi->port_id) || pi->uld_vis > 0) + if ((if_getcapenable(vi->ifp) & IFCAP_TOE) == 0) { + /* TOE is already disabled. */ return (0); - - KASSERT(uld_active(sc, ULD_TOM), - ("%s: TOM never initialized?", __func__)); - clrbit(&sc->offload_map, pi->port_id); + } + MPASS(isset(&sc->offload_map, pi->port_id)); + MPASS(pi->uld_vis > 0); + if (--pi->uld_vis == 0) + clrbit(&sc->offload_map, pi->port_id); } return (0); diff --git a/sys/dev/cxgbe/tom/t4_tom.c b/sys/dev/cxgbe/tom/t4_tom.c index d5b6b6ee2ce0..97cb380d0e71 100644 --- a/sys/dev/cxgbe/tom/t4_tom.c +++ b/sys/dev/cxgbe/tom/t4_tom.c @@ -1946,20 +1946,29 @@ done: static int t4_tom_deactivate(struct adapter *sc) { - int rc = 0; + int rc = 0, i, v; struct tom_data *td = sc->tom_softc; + struct vi_info *vi; ASSERT_SYNCHRONIZED_OP(sc); if (td == NULL) return (0); /* XXX. KASSERT? */ - if (sc->offload_map != 0) - return (EBUSY); /* at least one port has IFCAP_TOE enabled */ - if (uld_active(sc, ULD_IWARP) || uld_active(sc, ULD_ISCSI)) return (EBUSY); /* both iWARP and iSCSI rely on the TOE. */ + if (sc->offload_map != 0) { + for_each_port(sc, i) { + for_each_vi(sc->port[i], v, vi) { + toe_capability(vi, false); + if_setcapenablebit(vi->ifp, 0, IFCAP_TOE); + SETTOEDEV(vi->ifp, NULL); + } + } + MPASS(sc->offload_map == 0); + } + mtx_lock(&td->toep_list_lock); if (!TAILQ_EMPTY(&td->toep_list)) rc = EBUSY; @@ -2243,14 +2252,16 @@ t4_tom_mod_load(void) } static void -tom_uninit(struct adapter *sc, void *arg __unused) +tom_uninit(struct adapter *sc, void *arg) { + bool *ok_to_unload = arg; + if (begin_synchronized_op(sc, NULL, SLEEP_OK | INTR_OK, "t4tomun")) return; /* Try to free resources (works only if no port has IFCAP_TOE) */ - if (uld_active(sc, ULD_TOM)) - t4_deactivate_uld(sc, ULD_TOM); + if (uld_active(sc, ULD_TOM) && t4_deactivate_uld(sc, ULD_TOM) != 0) + *ok_to_unload = false; end_synchronized_op(sc, 0); } @@ -2258,7 +2269,11 @@ tom_uninit(struct adapter *sc, void *arg __unused) static int t4_tom_mod_unload(void) { - t4_iterate(tom_uninit, NULL); + bool ok_to_unload = true; + + t4_iterate(tom_uninit, &ok_to_unload); + if (!ok_to_unload) + return (EBUSY); if (t4_unregister_uld(&tom_uld_info, ULD_TOM) == EBUSY) return (EBUSY);