From nobody Fri Sep 09 17:12:34 2022 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MPMxb0Rb2z4bsr9; Fri, 9 Sep 2022 17:12:35 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MPMxZ6v5vz3C0J; Fri, 9 Sep 2022 17:12:34 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1662743555; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=tkbVj/1yrD1tHajaiJ83N4ERoj1edCvKgKcbrfX9Kts=; b=W411ywqnoiRsGc0PWFSCgXpfq32T3G9G/idZvi935Dt6nWyM9mNu0oRAvW2CyC2NzTQVc2 nQ5YOk0VNynQWZoo6oGgolUXag63gxMGbgY22OSam1dkieOz0MDQJ9pWpC3r0lpJDejGfD OaRCvBBhW1aRR33O6il7IIr0oy3ykWQX+vgjeAgBMmIVS9rKpn69vYRvreq/bp7MywV6Mw E4uCgdAr73hSYxWSJnWnkdYbRE37GmG9Kg2emEoByfU7duddWcxJ8sXwnme6vxwMxhmO0u kO2/Za5pOJ6R42QGZxNCvsrGrR1+i7Ylns8Oi7Asq2PS0gAWQ2+Vdqv7pu8dXA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MPMxZ5xhrz1Glw; Fri, 9 Sep 2022 17:12:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 289HCY2n045676; Fri, 9 Sep 2022 17:12:34 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 289HCYCi045675; Fri, 9 Sep 2022 17:12:34 GMT (envelope-from git) Date: Fri, 9 Sep 2022 17:12:34 GMT Message-Id: <202209091712.289HCYCi045675@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Doug Moore Subject: git: 412f27c18cdf - stable/13 - rb_tree: speed-up double rotation List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: dougm X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 412f27c18cdf5f3412e4445849e0b787d5437ece Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1662743555; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=tkbVj/1yrD1tHajaiJ83N4ERoj1edCvKgKcbrfX9Kts=; b=CD0S9XcfbYusVQkL8rb/02CgD/H+jvB5D/XP/uPZ6mUBfs4CFIOrF5vHcCOd3Vftwi57um whWMZWZMjPRw6aNZ+K0g/HFKAGOucOTrBS6769lx9fEwUMzyvpObB/j+9fa6gWqKVvyQsw fPUSdfgMf4CF3iKCAW32uXR5cL98ARzBUfCI/mNUId2pngPU3iMpJ0iP53ZqkaJx3G0bhB 03HaEz1iDsz7Yd/8wU3fjV7I1bbGjsx4MGX9tqpKaR42bHGFgHfusFNnj7QiytazSrqMHL mCt6Go6/1+fdZRsT/K5gRrcERamTKwalbjpLi16AkFS6prq9cijbDxDNA1tGBQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1662743555; a=rsa-sha256; cv=none; b=FeeOJmzk1mL7LgnU2KgWFc6g1BkGB/mpLDxXd3ZCWJskZxx2IkMomzD1jsZtxI2sx35TQw WX1zqsxKYjDL4J5vCybUZGk82xnXn+qL/n9WR0TjvIG6SXAwGc3rChbNn5yje4/o0prDMu ud12EixWCZcdAH03sKXmuLq1cOhiYxWzah2u4i4TypF2c+o/qy+bAzgpZ6qJ07NWwR6HXi eiMt/kieIxUCw/UuhNeP2YeH9qQEmBKToiH1ebnruqI8iQIJtjJ+6lHkqSSHaJTSDtWzMB YhUhzvuq67snua6zJtBVWQmQzMxVPnTnCmPj3xiJ+9KzAY+ThbFfB6ysP9deGQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by dougm: URL: https://cgit.FreeBSD.org/src/commit/?id=412f27c18cdf5f3412e4445849e0b787d5437ece commit 412f27c18cdf5f3412e4445849e0b787d5437ece Author: Doug Moore AuthorDate: 2022-08-19 23:11:29 +0000 Commit: Doug Moore CommitDate: 2022-09-09 17:11:44 +0000 rb_tree: speed-up double rotation RB_ROTATE_LEFT (and it symmetric twin) modify the rb-tree, adjusting pointers so that what started as a proper tree ends up a proper tree. When two consecutive rotations move the same node up the tree, some of the pointers changed in the first rotation are immediately changed again in the second - namely, the pointer from the rising node to its new parent, and the pointer from that parent back to the rising node. This change removes from RB_ROTATE macros the responsibility for managing those two pointers, and leaves it to the code that calls for rotations to fix up those pointers afterward. That drops a comparison and a pair of assignments from every INSERT_COLOR or REMOVE_COLOR call that ends in a double rotation. A side-effect of this change is that the SWAP_CHILD macro must take as a parameter a pointer to the node that is changing children, where it is now computed from the old child. Since this macro is called in a couple of places besides the RB_ROTATE macros, those calls are also affected. Reviewed by: alc MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D36266 (cherry picked from commit 02d0c43c9e53b3055b17719a184a813032040f79) --- sys/compat/linuxkpi/common/include/linux/rbtree.h | 3 +- sys/sys/tree.h | 63 ++++++++++++++--------- 2 files changed, 41 insertions(+), 25 deletions(-) diff --git a/sys/compat/linuxkpi/common/include/linux/rbtree.h b/sys/compat/linuxkpi/common/include/linux/rbtree.h index de5ef0d6a3ce..1db9b1d4fa21 100644 --- a/sys/compat/linuxkpi/common/include/linux/rbtree.h +++ b/sys/compat/linuxkpi/common/include/linux/rbtree.h @@ -132,7 +132,8 @@ rb_replace_node(struct rb_node *victim, struct rb_node *new, struct rb_root *root) { - RB_SWAP_CHILD((struct linux_root *)root, victim, new, __entry); + RB_SWAP_CHILD((struct linux_root *)root, rb_parent(victim), + victim, new, __entry); if (victim->rb_left) RB_SET_PARENT(victim->rb_left, new, __entry); if (victim->rb_right) diff --git a/sys/sys/tree.h b/sys/sys/tree.h index 50c4ec980b1e..db5de3b8ec87 100644 --- a/sys/sys/tree.h +++ b/sys/sys/tree.h @@ -378,31 +378,40 @@ struct { \ } while ((rb_update_tmp = RB_PARENT(rb_update_tmp, field)) != NULL); \ } while (0) -#define RB_SWAP_CHILD(head, out, in, field) do { \ - if (RB_PARENT(out, field) == NULL) \ +#define RB_SWAP_CHILD(head, par, out, in, field) do { \ + if (par == NULL) \ RB_ROOT(head) = (in); \ - else if ((out) == RB_LEFT(RB_PARENT(out, field), field)) \ - RB_LEFT(RB_PARENT(out, field), field) = (in); \ + else if ((out) == RB_LEFT(par, field)) \ + RB_LEFT(par, field) = (in); \ else \ - RB_RIGHT(RB_PARENT(out, field), field) = (in); \ + RB_RIGHT(par, field) = (in); \ } while (/*CONSTCOND*/ 0) -#define RB_ROTATE_LEFT(head, elm, tmp, field) do { \ +/* + * RB_ROTATE macros partially restructure the tree to improve + * balance. The ROTATE_RIGHT case is just a reflection of the + * ROTATE_LEFT case. Initially, tmp is a right child of elm. After + * rotation, elm is a left child of tmp, and the subtree that + * represented the items between them, which formerly hung to the left + * of tmp now hangs to the right of elm. The parent-child + * relationship between elm and its former parent is not changed; + * where this macro once updated those fields, that is now left to the + * caller of RB_ROTATE to clean up, so that a pair of rotations does + * not twice update the came pair of pointer fields with distinct + * values. + */ +#define RB_ROTATE_LEFT(elm, tmp, field) do { \ if ((RB_RIGHT(elm, field) = RB_LEFT(tmp, field)) != NULL) { \ RB_SET_PARENT(RB_RIGHT(elm, field), elm, field); \ } \ - RB_SET_PARENT(tmp, RB_PARENT(elm, field), field); \ - RB_SWAP_CHILD(head, elm, tmp, field); \ RB_LEFT(tmp, field) = (elm); \ RB_SET_PARENT(elm, tmp, field); \ } while (/*CONSTCOND*/ 0) -#define RB_ROTATE_RIGHT(head, elm, tmp, field) do { \ +#define RB_ROTATE_RIGHT(elm, tmp, field) do { \ if ((RB_LEFT(elm, field) = RB_RIGHT(tmp, field)) != NULL) { \ RB_SET_PARENT(RB_LEFT(elm, field), elm, field); \ } \ - RB_SET_PARENT(tmp, RB_PARENT(elm, field), field); \ - RB_SWAP_CHILD(head, elm, tmp, field); \ RB_RIGHT(tmp, field) = (elm); \ RB_SET_PARENT(elm, tmp, field); \ } while (/*CONSTCOND*/ 0) @@ -479,8 +488,9 @@ name##_RB_INSERT_COLOR(struct name *head, struct type *elm) \ * when a value has been assigned to 'child' in the previous \ * one. \ */ \ - struct type *child, *parent; \ + struct type *child, *gpar, *parent; \ while ((parent = RB_PARENT(elm, field)) != NULL) { \ + gpar = RB_PARENT(parent, field); \ if (RB_LEFT(parent, field) == elm) { \ if (RB_RED_LEFT(parent, field)) { \ RB_FLIP_LEFT(parent, field); \ @@ -496,7 +506,7 @@ name##_RB_INSERT_COLOR(struct name *head, struct type *elm) \ child = elm; \ else { \ /* coverity[uninit_use] */ \ - RB_ROTATE_LEFT(head, elm, child, field);\ + RB_ROTATE_LEFT(elm, child, field); \ if (RB_RED_RIGHT(child, field)) \ RB_FLIP_LEFT(parent, field); \ if (RB_RED_LEFT(child, field)) \ @@ -507,7 +517,7 @@ name##_RB_INSERT_COLOR(struct name *head, struct type *elm) \ RB_RED_MASK) == 0) \ elm = child; \ } \ - RB_ROTATE_RIGHT(head, parent, child, field); \ + RB_ROTATE_RIGHT(parent, child, field); \ } else { \ if (RB_RED_RIGHT(parent, field)) { \ RB_FLIP_RIGHT(parent, field); \ @@ -523,7 +533,7 @@ name##_RB_INSERT_COLOR(struct name *head, struct type *elm) \ child = elm; \ else { \ /* coverity[uninit_use] */ \ - RB_ROTATE_RIGHT(head, elm, child, field);\ + RB_ROTATE_RIGHT(elm, child, field); \ if (RB_RED_LEFT(child, field)) \ RB_FLIP_RIGHT(parent, field); \ if (RB_RED_RIGHT(child, field)) \ @@ -534,8 +544,10 @@ name##_RB_INSERT_COLOR(struct name *head, struct type *elm) \ RB_RED_MASK) == 0) \ elm = child; \ } \ - RB_ROTATE_LEFT(head, parent, child, field); \ + RB_ROTATE_LEFT(parent, child, field); \ } \ + RB_SET_PARENT(child, gpar, field); \ + RB_SWAP_CHILD(head, gpar, parent, child, field); \ RB_BITS(child, field) &= ~RB_RED_MASK; \ if (elm != child) \ RB_AUGMENT(elm); \ @@ -560,7 +572,7 @@ attr void \ name##_RB_REMOVE_COLOR(struct name *head, \ struct type *parent, struct type *elm) \ { \ - struct type *sib; \ + struct type *gpar, *sib; \ if (RB_LEFT(parent, field) == elm && \ RB_RIGHT(parent, field) == elm) { \ RB_BITS(parent, field) &= ~RB_RED_MASK; \ @@ -570,6 +582,7 @@ name##_RB_REMOVE_COLOR(struct name *head, \ return; \ } \ do { \ + gpar = RB_PARENT(parent, field); \ if (RB_LEFT(parent, field) == elm) { \ if (!RB_RED_LEFT(parent, field)) { \ RB_FLIP_LEFT(parent, field); \ @@ -588,7 +601,7 @@ name##_RB_REMOVE_COLOR(struct name *head, \ continue; \ case RB_RED_R: \ elm = RB_LEFT(sib, field); \ - RB_ROTATE_RIGHT(head, sib, elm, field); \ + RB_ROTATE_RIGHT(sib, elm, field); \ if (RB_RED_LEFT(elm, field)) \ RB_FLIP_ALL(parent, field); \ else \ @@ -613,7 +626,7 @@ name##_RB_REMOVE_COLOR(struct name *head, \ elm = sib; \ break; \ } \ - RB_ROTATE_LEFT(head, parent, elm, field); \ + RB_ROTATE_LEFT(parent, elm, field); \ } else { \ if (!RB_RED_RIGHT(parent, field)) { \ RB_FLIP_RIGHT(parent, field); \ @@ -632,7 +645,7 @@ name##_RB_REMOVE_COLOR(struct name *head, \ continue; \ case RB_RED_L: \ elm = RB_RIGHT(sib, field); \ - RB_ROTATE_LEFT(head, sib, elm, field); \ + RB_ROTATE_LEFT(sib, elm, field); \ if (RB_RED_RIGHT(elm, field)) \ RB_FLIP_ALL(parent, field); \ else \ @@ -657,8 +670,10 @@ name##_RB_REMOVE_COLOR(struct name *head, \ elm = sib; \ break; \ } \ - RB_ROTATE_RIGHT(head, parent, elm, field); \ + RB_ROTATE_RIGHT(parent, elm, field); \ } \ + RB_SET_PARENT(elm, gpar, field); \ + RB_SWAP_CHILD(head, gpar, parent, elm, field); \ if (sib != elm) \ RB_AUGMENT(sib); \ break; \ @@ -669,10 +684,10 @@ name##_RB_REMOVE_COLOR(struct name *head, \ attr struct type * \ name##_RB_REMOVE(struct name *head, struct type *elm) \ { \ - struct type *child, *old, *parent, *right; \ + struct type *child, *gpar, *old, *parent, *right; \ \ old = elm; \ - parent = RB_PARENT(elm, field); \ + gpar = parent = RB_PARENT(elm, field); \ right = RB_RIGHT(elm, field); \ if (RB_LEFT(elm, field) == NULL) \ elm = child = right; \ @@ -695,7 +710,7 @@ name##_RB_REMOVE(struct name *head, struct type *elm) \ RB_SET_PARENT(RB_LEFT(old, field), elm, field); \ elm->field = old->field; \ } \ - RB_SWAP_CHILD(head, old, elm, field); \ + RB_SWAP_CHILD(head, gpar, old, elm, field); \ if (child != NULL) \ RB_SET_PARENT(child, parent, field); \ if (parent != NULL) { \