From nobody Sun Sep 15 06:07:33 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X5yJ62Ykrz5WY4V; Sun, 15 Sep 2024 06:07:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X5yJ60D4Cz4Zmq; Sun, 15 Sep 2024 06:07:34 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726380454; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Tf0HJZja5FlcexAPJAhnlee7QfEu6Zwf1TpCmFcxJnU=; b=szF5WM8OxBz4NAfR9LmJCEh+RZ8LyfalvwnEgphubbadqxqlT375BVjjD2IjksY5t6wKmN gc5MBpOUpmZjnUYTEZ93zXYlCPdjND6K9BP8nXknNC+B8ux7+W4J0zkCIyM5/Zi5AA69fY ty3iSqqWvIa9QIswatmRm7aWkxXrmbQA4NhaS8qVPBWVacz6yP2SJRq89rep5EJgeykwJw Br8lWIKJvc1my1kBgP1A5XLX+Wqm3y36vUyu1FycpXkkUSs9smx1pYD4gyaZ9ihFQ4/TKo loFCCcpIJaqJ8hP4lK9vTTqZyv/7+xMtHYIWYQdoE60Ds7EmpOrT6Z5ccPVaQQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726380454; a=rsa-sha256; cv=none; b=nAGJuYHwo8HdZ2ng8frwGN9AuieD9CshNMLf0l7jIEht4Oi3PdjakZF2UBxSyog9AMk7OB GmovZfltSqASD3agUpdUeY04YM43QUxcFsBgWBmp2JIemLEvmhdTSVseFZGEYWpbGpZGOx RGVSSXNPCrPJq3Z2cuBP8inZodJ+VMOQ4FGP29xTvTSLYbzzJAUnnM3hOcUfkNEZJf/5oR dUw+0OEHcwM/GiErxBdgtSJJwA2JA9TjEMUs3uRF7x46CRew0jcBz1HzTJNR8XPg7pQPm9 fBb1z6oC9seZGVXjmHUd9zUFveIjDF6oLStu7+Q9mJdrdYGiHvsHhk69ijajTg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726380454; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Tf0HJZja5FlcexAPJAhnlee7QfEu6Zwf1TpCmFcxJnU=; b=h/qPPtbQArJzL7R5HHicjnea71lRYNQ2ujlH9HHRQpOz4UtwTAfUV7Z62QzjVsS5oef0Ig Mr2QBUh/sBIhLXBD3RsN6InZ1bQ1EPPUpnp4unNPSjLdvDC+gUwNuNwwsdTuD5bpZ2CvLB O9cxVIfjPKsJ1ocz/C2K+lo0CZ3qs+VBWjCjRnTVkkZcxN6HTTS9LSCaaCKYQP2lRz/zWD m0v5lrrycpX7YWjtHmwWrTG7xxp9+/IhWazenUXR1PdjZazsuPEst8Qhsyo00kUVuZcsTz AFgxJYfAWv5VRmy5eyyJNQ+Kr4YYSxlWoW+nnhTLYvBiPH6AqeQKUEbBVF9F3A== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X5yJ56NKgzyl3; Sun, 15 Sep 2024 06:07:33 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 48F67XWi029319; Sun, 15 Sep 2024 06:07:33 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 48F67XXO029316; Sun, 15 Sep 2024 06:07:33 GMT (envelope-from git) Date: Sun, 15 Sep 2024 06:07:33 GMT Message-Id: <202409150607.48F67XXO029316@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Doug Moore Subject: git: 4ccad5452058 - main - swap_pager: avoid meta_transfer race List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: dougm X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 4ccad5452058749a2a6ee5865ff485a3f11b20df Auto-Submitted: auto-generated The branch main has been updated by dougm: URL: https://cgit.FreeBSD.org/src/commit/?id=4ccad5452058749a2a6ee5865ff485a3f11b20df commit 4ccad5452058749a2a6ee5865ff485a3f11b20df Author: Doug Moore AuthorDate: 2024-09-15 06:00:00 +0000 Commit: Doug Moore CommitDate: 2024-09-15 06:00:00 +0000 swap_pager: avoid meta_transfer race Function swp_pager_meta_transfer expects that after dropping and reacquiring an object lock, the swap block it's processing still exists, and has not been removed from the trie and freed. Rewrite to avoid depending on that, by scrubbing, removing and freeing it before releasing the lock. Reviewed by: alc, markj Differential Revision: https://reviews.freebsd.org/D46629 --- sys/vm/swap_pager.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/sys/vm/swap_pager.c b/sys/vm/swap_pager.c index c7a9f16a2953..64ae4ad78786 100644 --- a/sys/vm/swap_pager.c +++ b/sys/vm/swap_pager.c @@ -2199,9 +2199,11 @@ swp_pager_meta_transfer(vm_object_t srcobject, vm_object_t dstobject, { struct page_range range; struct swblk *sb; - daddr_t blk; + daddr_t blk, d[SWAP_META_PAGES]; vm_pindex_t offset, last; - int i, limit, start; + int d_mask, i, limit, start; + _Static_assert(8 * sizeof(d_mask) >= SWAP_META_PAGES, + "d_mask not big enough"); VM_OBJECT_ASSERT_WLOCKED(srcobject); VM_OBJECT_ASSERT_WLOCKED(dstobject); @@ -2210,39 +2212,50 @@ swp_pager_meta_transfer(vm_object_t srcobject, vm_object_t dstobject, return; swp_pager_init_freerange(&range); + d_mask = 0; offset = pindex; last = pindex + count; sb = swblk_start_limit(srcobject, pindex, last); start = (sb != NULL && sb->p < pindex) ? pindex - sb->p : 0; for (; sb != NULL; sb = swblk_start_limit(srcobject, pindex, last), start = 0) { - limit = MIN(last - sb->p, SWAP_META_PAGES); + pindex = sb->p; + MPASS(d_mask == 0); + limit = MIN(last - pindex, SWAP_META_PAGES); for (i = start; i < limit; i++) { if (sb->d[i] == SWAPBLK_NONE) continue; blk = swp_pager_meta_build(dstobject, - sb->p + i - offset, sb->d[i], true); + pindex + i - offset, sb->d[i], true); if (blk == sb->d[i]) { /* - * Destination has no swapblk and is not - * resident, so transfer source. - * swp_pager_meta_build() failed memory - * allocation already, likely to sleep in retry. + * Failed memory allocation stopped transfer; + * save this block for transfer with lock + * released. */ - VM_OBJECT_WUNLOCK(srcobject); - swp_pager_meta_build(dstobject, - sb->p + i - offset, sb->d[i], false); - VM_OBJECT_WLOCK(srcobject); + d[i] = blk; + d_mask |= 1 << i; } else if (blk != SWAPBLK_NONE) swp_pager_update_freerange(&range, sb->d[i]); sb->d[i] = SWAPBLK_NONE; } - pindex = sb->p + SWAP_META_PAGES; if (swp_pager_swblk_empty(sb, 0, start) && swp_pager_swblk_empty(sb, limit, SWAP_META_PAGES)) { swblk_lookup_remove(srcobject, sb); uma_zfree(swblk_zone, sb); } + if (d_mask != 0) { + /* Finish block transfer, with the lock released. */ + VM_OBJECT_WUNLOCK(srcobject); + do { + i = ffs(d_mask) - 1; + swp_pager_meta_build(dstobject, + pindex + i - offset, d[i], false); + d_mask &= ~(1 << i); + } while (d_mask != 0); + VM_OBJECT_WLOCK(srcobject); + } + pindex += SWAP_META_PAGES; } swp_pager_freeswapspace(&range); }