From nobody Mon Nov 15 14:07:37 2021 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 513E91896AB7; Mon, 15 Nov 2021 14:07:37 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ht9xj1vsTz4c3y; Mon, 15 Nov 2021 14:07:37 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 213C81AE61; Mon, 15 Nov 2021 14:07:37 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1AFE7bYm006270; Mon, 15 Nov 2021 14:07:37 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1AFE7b7E006269; Mon, 15 Nov 2021 14:07:37 GMT (envelope-from git) Date: Mon, 15 Nov 2021 14:07:37 GMT Message-Id: <202111151407.1AFE7b7E006269@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: d5ebaa6f8f85 - stable/13 - uma: Improve M_USE_RESERVE handling in keg_fetch_slab() List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: d5ebaa6f8f850bb6f6273f01386832efcb295827 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=d5ebaa6f8f850bb6f6273f01386832efcb295827 commit d5ebaa6f8f850bb6f6273f01386832efcb295827 Author: Mark Johnston AuthorDate: 2021-11-01 13:27:35 +0000 Commit: Mark Johnston CommitDate: 2021-11-15 14:06:54 +0000 uma: Improve M_USE_RESERVE handling in keg_fetch_slab() M_USE_RESERVE is used in a couple of places in the VM to avoid unbounded recursion when the direct map is not available, as is the case on 32-bit platforms or when certain kernel sanitizers (KASAN and KMSAN) are enabled. For example, to allocate KVA, the kernel might allocate a kernel map entry, which might require a new slab, which requires KVA. For these zones, we use uma_prealloc() to populate a reserve of items, and then in certain serialized contexts M_USE_RESERVE can be used to guarantee a successful allocation. uma_prealloc() allocates the requested number of items, distributing them evenly among NUMA domains. Thus, in a first-touch zone, to satisfy an M_USE_RESERVE allocation we might have to check the slab lists of other domains than the current one to provide the semantics expected by consumers. So, try harder to find an item if M_USE_RESERVE is specified and the keg doesn't have anything for current (first-touch) domain. Specifically, fall back to a round-robin slab allocation. This change fixes boot-time panics on NUMA systems with KASAN or KMSAN enabled.[1] Alternately we could have uma_prealloc() allocate the requested number of items for each domain, but for some existing consumers this would be quite wasteful. In general I think keg_fetch_slab() should try harder to find free slabs in other domains before trying to allocate fresh ones, but let's limit this to M_USE_RESERVE for now. Also fix a separate problem that I noticed: in a non-round-robin slab allocation with M_WAITOK, rather than sleeping after a failed slab allocation we simply try again. Call vm_wait_domain() before retrying. Reported by: mjg, tuexen [1] Reviewed by: alc Sponsored by: The FreeBSD Foundation (cherry picked from commit fab343a7168a2f033073bb5f65b5af17d9092c6f) --- sys/vm/uma_core.c | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/sys/vm/uma_core.c b/sys/vm/uma_core.c index 3e4e3c7c4ce1..1fb066d71762 100644 --- a/sys/vm/uma_core.c +++ b/sys/vm/uma_core.c @@ -3858,6 +3858,9 @@ keg_fetch_slab(uma_keg_t keg, uma_zone_t zone, int rdomain, const int flags) int aflags, domain; bool rr; + KASSERT((flags & (M_WAITOK | M_NOVM)) != (M_WAITOK | M_NOVM), + ("%s: invalid flags %#x", __func__, flags)); + restart: /* * Use the keg's policy if upper layers haven't already specified a @@ -3883,17 +3886,29 @@ restart: return (slab); /* - * M_NOVM means don't ask at all! + * M_NOVM is used to break the recursion that can otherwise + * occur if low-level memory management routines use UMA. */ - if (flags & M_NOVM) - break; + if ((flags & M_NOVM) == 0) { + slab = keg_alloc_slab(keg, zone, domain, flags, aflags); + if (slab != NULL) + return (slab); + } - slab = keg_alloc_slab(keg, zone, domain, flags, aflags); - if (slab != NULL) - return (slab); - if (!rr && (flags & M_WAITOK) == 0) - break; - if (rr && vm_domainset_iter_policy(&di, &domain) != 0) { + if (!rr) { + if ((flags & M_USE_RESERVE) != 0) { + /* + * Drain reserves from other domains before + * giving up or sleeping. It may be useful to + * support per-domain reserves eventually. + */ + rdomain = UMA_ANYDOMAIN; + goto restart; + } + if ((flags & M_WAITOK) == 0) + break; + vm_wait_domain(domain); + } else if (vm_domainset_iter_policy(&di, &domain) != 0) { if ((flags & M_WAITOK) != 0) { vm_wait_doms(&keg->uk_dr.dr_policy->ds_mask, 0); goto restart;