From nobody Tue Sep 27 15:15:21 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4McNV158mjz4V2gK; Tue, 27 Sep 2022 15:15:21 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4McNV14dlYz3fsH; Tue, 27 Sep 2022 15:15:21 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1664291721; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DDdMpfeEUFvtN1DskoFdETUdFyRX4E3sQb594KsqzXo=; b=P4kGaQ+ZJl7ua6+G1ZykArmbRG5d7HbTJtzE43kbuf5jY6Gkvhwilr25hyP8pXX167u3nt 8/9g6DtzWMeL09xkTSueUu3zhWXBkEKWAMOHf4S9gM4YmSRG3gF6EAvOtrEZ1p4WjbdtQd rG5/hxnBot4kot3+tfkwODQl73Ou1LXlkd/LhJMz5K8mjVKn2d8OKEWNXIMdP5FFvekmDW IcS4G5YEqQBahpSagPLFwQnRJbgjwWmQCg8LsnRHP+GZ4A3KxCl14qcL+MKn4JNrbi7P/H 6qvQkxohaeRDHd540tAy3InSRJwMEiXykkpYz0vDaCopt4XEVWATuEYiboVs4A== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4McNV13bNMz18wm; Tue, 27 Sep 2022 15:15:21 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 28RFFLbd015535; Tue, 27 Sep 2022 15:15:21 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 28RFFLpw015534; Tue, 27 Sep 2022 15:15:21 GMT (envelope-from git) Date: Tue, 27 Sep 2022 15:15:21 GMT Message-Id: <202209271515.28RFFLpw015534@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Andrew Turner Subject: git: 92d73b0b2564 - main - arm64 pmap: batch chunk removal in pmap_remove_pages List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: andrew X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 92d73b0b256465adff6722a817220193b6037448 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1664291721; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DDdMpfeEUFvtN1DskoFdETUdFyRX4E3sQb594KsqzXo=; b=Ft5uB3o4bOM4oQh8pXhYTxt88saF5+2V+iykWz1Z6cgEP7ysQhAm41QdWcguUlp32+FUeG wSLvnX6wJ5ttVineolN6M3gKT3qIW84kqZKC+SlsQBrY5YieFsXukft17Bg9iSHGBHzSo5 gdrptb6rbR4zBmdDRh2m0z2K0jbjwJdGg3aQSQehlXFXi3AgXYqbNX32SteOYPSiZ+81PF VJWJbRAYxQ7SUmig1lvGOMYHISZeA1yzKlZXk7Mkf/oNzGfOfVmLrLqlz74svFXjjoAdKm ULIdzDKeNOxjEje8TQlOb4J2EpzCAVF5PYxwKWayhqAsQ2GJklLWhbPh3YaXqA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1664291721; a=rsa-sha256; cv=none; b=wqJP4zxyxUMok0901IHcuJE/G2qnC0dWcNxfT2psYEzTe7ga55IY/MrUUIzmSuqbBuAmJa uZH4CJH4//IWSbLES5yOMv7HG/je77Y3wZM3DFBdSfh+XWK1WUM2cFSWqwyNzA06seTk97 BAS60FhIK7UNGMfdD9PGbmvhXTVuMvsM2ZUkZr5lAi+h2uNjOnVbX0qb6OFy9B5RlYA9d/ e1hYiJTOh2GELtH13I33OX6mevOjpxqQTDyt0Pt9avXnTyeLteWbSyGvYPl+p/fDP1pU9I EmFOpN3aqOypVMGcjta9KlbO8mEABykhn4Ol5/p1ahevBkRBNTBGAJwbtoUI5g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by andrew: URL: https://cgit.FreeBSD.org/src/commit/?id=92d73b0b256465adff6722a817220193b6037448 commit 92d73b0b256465adff6722a817220193b6037448 Author: Andrew Turner AuthorDate: 2022-08-23 09:50:18 +0000 Commit: Andrew Turner CommitDate: 2022-09-27 15:03:08 +0000 arm64 pmap: batch chunk removal in pmap_remove_pages As with amd64 batch chunk removal in pmap_remove_pages to move it out of the pv list lock. This is one of the main contested locks when running poudriere on a 160 core Ampere Altra server. Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D36305 --- sys/arm64/arm64/pmap.c | 41 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 5 deletions(-) diff --git a/sys/arm64/arm64/pmap.c b/sys/arm64/arm64/pmap.c index dce52034f27c..0c2f623aa9a3 100644 --- a/sys/arm64/arm64/pmap.c +++ b/sys/arm64/arm64/pmap.c @@ -380,7 +380,10 @@ SYSCTL_INT(_vm_pmap, OID_AUTO, superpages_enabled, #define PMAP_ENTER_NORECLAIM 0x1000000 /* Don't reclaim PV entries. */ #define PMAP_ENTER_NOREPLACE 0x2000000 /* Don't replace mappings. */ +TAILQ_HEAD(pv_chunklist, pv_chunk); + static void free_pv_chunk(struct pv_chunk *pc); +static void free_pv_chunk_batch(struct pv_chunklist *batch); static void free_pv_entry(pmap_t pmap, pv_entry_t pv); static pv_entry_t get_pv_entry(pmap_t pmap, struct rwlock **lockp); static vm_page_t reclaim_pv_chunk(pmap_t locked_pmap, struct rwlock **lockp); @@ -2756,13 +2759,10 @@ free_pv_entry(pmap_t pmap, pv_entry_t pv) } static void -free_pv_chunk(struct pv_chunk *pc) +free_pv_chunk_dequeued(struct pv_chunk *pc) { vm_page_t m; - mtx_lock(&pv_chunks_mutex); - TAILQ_REMOVE(&pv_chunks, pc, pc_lru); - mtx_unlock(&pv_chunks_mutex); PV_STAT(atomic_subtract_int(&pv_entry_spare, _NPCPV)); PV_STAT(atomic_subtract_int(&pc_chunk_count, 1)); PV_STAT(atomic_add_int(&pc_chunk_frees, 1)); @@ -2773,6 +2773,34 @@ free_pv_chunk(struct pv_chunk *pc) vm_page_free(m); } +static void +free_pv_chunk(struct pv_chunk *pc) +{ + mtx_lock(&pv_chunks_mutex); + TAILQ_REMOVE(&pv_chunks, pc, pc_lru); + mtx_unlock(&pv_chunks_mutex); + free_pv_chunk_dequeued(pc); +} + +static void +free_pv_chunk_batch(struct pv_chunklist *batch) +{ + struct pv_chunk *pc, *npc; + + if (TAILQ_EMPTY(batch)) + return; + + mtx_lock(&pv_chunks_mutex); + TAILQ_FOREACH(pc, batch, pc_list) { + TAILQ_REMOVE(&pv_chunks, pc, pc_lru); + } + mtx_unlock(&pv_chunks_mutex); + + TAILQ_FOREACH_SAFE(pc, batch, pc_list, npc) { + free_pv_chunk_dequeued(pc); + } +} + /* * Returns a new PV entry, allocating a new PV chunk from the system when * needed. If this PV chunk allocation fails and a PV list lock pointer was @@ -5248,6 +5276,7 @@ pmap_remove_pages(pmap_t pmap) pd_entry_t *pde; pt_entry_t *pte, tpte; struct spglist free; + struct pv_chunklist free_chunks; vm_page_t m, ml3, mt; pv_entry_t pv; struct md_page *pvh; @@ -5260,6 +5289,7 @@ pmap_remove_pages(pmap_t pmap) lock = NULL; + TAILQ_INIT(&free_chunks); SLIST_INIT(&free); PMAP_LOCK(pmap); TAILQ_FOREACH_SAFE(pc, &pmap->pm_pvchunk, pc_list, npc) { @@ -5400,12 +5430,13 @@ pmap_remove_pages(pmap_t pmap) PV_STAT(atomic_subtract_long(&pv_entry_count, freed)); if (allfree) { TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); - free_pv_chunk(pc); + TAILQ_INSERT_TAIL(&free_chunks, pc, pc_list); } } if (lock != NULL) rw_wunlock(lock); pmap_invalidate_all(pmap); + free_pv_chunk_batch(&free_chunks); PMAP_UNLOCK(pmap); vm_page_free_pages_toq(&free, true); }