From nobody Wed Dec 14 19:43:49 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NXQln71R9z4kPyB; Wed, 14 Dec 2022 19:43:49 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NXQln6WgRz3xNL; Wed, 14 Dec 2022 19:43:49 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671047029; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Zn6pbJqCkcc7NZ1BvcQfhYzEhRFgxAtngJvL9IzDJNo=; b=oDOOwT40D48kfWCZmEto8/CWeD/9ISWlZkXbUP7pw5gOXUr9pQYzhzhXKl34LXCghApOnL bxxvtIlxOa77YRVbW0r9BHlGC5pMk5qN7VHdaG80p4Y423aVolNQmRsed1SpOXEFTeaH// Fog7grjMtuS8mMMdA0uN+aaLKf1tkjsxjyeWF02uKJTEyC2VGKu1io3xPKjeY3Eb5HbWXU NJeamAIKc1wFElW83MuLLzd/qM+WK8fOLcu04Nhppu3+VnFbzfnKzQcyGbNhmBfYqHbGjb i4ZEOdXQlDjO+AulqkFA46g+aw0Lc54qSYnNv3eL9cNBpufzv4bMxZlKd3bE0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671047029; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Zn6pbJqCkcc7NZ1BvcQfhYzEhRFgxAtngJvL9IzDJNo=; b=VEmKiE/qQOCYrH/7Tnuoug7Vw0Q2mchcbvpkFYSkPnPo+GtoqfDfwAfH+csa4VDfVmzcLh N8yrrBZY8Q0Vi8p8D1Z7W/sGAINh1S3OsDSpypOM2MuzyKDE6saYuBr5e3W67n0w+tYG/x 9B5Ut22N/Fe5xPybUq6b0lxIXA7tv/F5ljUmAg97IsC9sIefiMLLVlCUMNiYzGTCy4hmVF gMB/KsTL5e9q9NR/DNGkFqoDy54yykT8lTmMP2uHyP0oY+lq9/zcv8T+mMICCtzQ7sO+A4 TIR7vluk88b6PfW3SBWe9Sgo2sgvTr72WWIcEKoZcNWph++U+ej60lO1AdKJnw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671047029; a=rsa-sha256; cv=none; b=uTetwyy2i0jC7uAMfvIdAkikfTcwlsl++gSavGuT90tvTap6YeaspzdOdf373XGfgaCq6o Eczx7zonyOjua6UzFxj4ElJjDHFOYvyZPTxQDfGaqxy+9m1LIIG++u4RYI4E0NcjZJ/LOf JN+H1sIorBcsS33E9Yk4A1HyCkqTlgu/QWu7xI/ScAAMusZIM0UBtvzRGuommk1L21LGaN QbI3VJzkCCeuiQSJ2M4SdhYOC+iyQdqLKta5e6hgKrtmPPQcHEpw2+ucfKXSuTEYY77x2W 9M9m29WFm+W4YAm9xcA9OUQsdlF0e7AXhjpn+en/ZqvUOTNi0R4PQb9LWTO/hQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4NXQln5bK2z15js; Wed, 14 Dec 2022 19:43:49 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 2BEJhnNC035649; Wed, 14 Dec 2022 19:43:49 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 2BEJhnkW035648; Wed, 14 Dec 2022 19:43:49 GMT (envelope-from git) Date: Wed, 14 Dec 2022 19:43:49 GMT Message-Id: <202212141943.2BEJhnkW035648@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Andrew Gallatin Subject: git: 1cac76c93fb7 - main - vm: reduce lock contention when processing vm batchqueues List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: gallatin X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 1cac76c93fb7f627fd9e304cbd99e8c8a2b8fce8 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by gallatin: URL: https://cgit.FreeBSD.org/src/commit/?id=1cac76c93fb7f627fd9e304cbd99e8c8a2b8fce8 commit 1cac76c93fb7f627fd9e304cbd99e8c8a2b8fce8 Author: Andrew Gallatin AuthorDate: 2022-12-14 19:34:07 +0000 Commit: Andrew Gallatin CommitDate: 2022-12-14 19:34:07 +0000 vm: reduce lock contention when processing vm batchqueues Rather than waiting until the batchqueue is full to acquire the lock & process the queue, we now start trying to acquire the lock using trylocks when the batchqueue is 1/2 full. This removes almost all contention on the vm pagequeue mutex for for our busy sendfile() based web workload. It also greadly reduces the amount of time a network driver ithread remains blocked on a mutex, and eliminates some packet drops under heavy load. So that the system does not loose the benefit of processing large batchqueues, I've doubled the size of the batchqueues. This way, when there is no contention, we process the same batch size as before. This has been run for several months on a busy Netflix server, as well as on my personal desktop. Reviewed by: markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D37305 --- sys/amd64/include/vmparam.h | 2 +- sys/powerpc/include/vmparam.h | 2 +- sys/vm/vm_page.c | 17 +++++++++++++++-- sys/vm/vm_pageout.c | 2 +- sys/vm/vm_pagequeue.h | 12 +++++++----- 5 files changed, 25 insertions(+), 10 deletions(-) diff --git a/sys/amd64/include/vmparam.h b/sys/amd64/include/vmparam.h index fc88296f754c..205848489644 100644 --- a/sys/amd64/include/vmparam.h +++ b/sys/amd64/include/vmparam.h @@ -293,7 +293,7 @@ * Use a fairly large batch size since we expect amd64 systems to have lots of * memory. */ -#define VM_BATCHQUEUE_SIZE 31 +#define VM_BATCHQUEUE_SIZE 63 /* * The pmap can create non-transparent large page mappings. diff --git a/sys/powerpc/include/vmparam.h b/sys/powerpc/include/vmparam.h index 77457717a3fd..1b9873aede4a 100644 --- a/sys/powerpc/include/vmparam.h +++ b/sys/powerpc/include/vmparam.h @@ -263,7 +263,7 @@ extern int vm_level_0_order; * memory. */ #ifdef __powerpc64__ -#define VM_BATCHQUEUE_SIZE 31 +#define VM_BATCHQUEUE_SIZE 63 #endif /* diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 2b7bc6a5b66e..797207205f42 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -3662,19 +3662,32 @@ vm_page_pqbatch_submit(vm_page_t m, uint8_t queue) { struct vm_batchqueue *bq; struct vm_pagequeue *pq; - int domain; + int domain, slots_remaining; KASSERT(queue < PQ_COUNT, ("invalid queue %d", queue)); domain = vm_page_domain(m); critical_enter(); bq = DPCPU_PTR(pqbatch[domain][queue]); - if (vm_batchqueue_insert(bq, m)) { + slots_remaining = vm_batchqueue_insert(bq, m); + if (slots_remaining > (VM_BATCHQUEUE_SIZE >> 1)) { + /* keep building the bq */ + critical_exit(); + return; + } else if (slots_remaining > 0 ) { + /* Try to process the bq if we can get the lock */ + pq = &VM_DOMAIN(domain)->vmd_pagequeues[queue]; + if (vm_pagequeue_trylock(pq)) { + vm_pqbatch_process(pq, bq, queue); + vm_pagequeue_unlock(pq); + } critical_exit(); return; } critical_exit(); + /* if we make it here, the bq is full so wait for the lock */ + pq = &VM_DOMAIN(domain)->vmd_pagequeues[queue]; vm_pagequeue_lock(pq); critical_enter(); diff --git a/sys/vm/vm_pageout.c b/sys/vm/vm_pageout.c index bb12a7e335d5..2945b53835c6 100644 --- a/sys/vm/vm_pageout.c +++ b/sys/vm/vm_pageout.c @@ -1405,7 +1405,7 @@ vm_pageout_reinsert_inactive(struct scan_state *ss, struct vm_batchqueue *bq, pq = ss->pq; if (m != NULL) { - if (vm_batchqueue_insert(bq, m)) + if (vm_batchqueue_insert(bq, m) != 0) return; vm_pagequeue_lock(pq); delta += vm_pageout_reinsert_inactive_page(pq, marker, m); diff --git a/sys/vm/vm_pagequeue.h b/sys/vm/vm_pagequeue.h index a9d4c920e5be..268d53a391db 100644 --- a/sys/vm/vm_pagequeue.h +++ b/sys/vm/vm_pagequeue.h @@ -75,7 +75,7 @@ struct vm_pagequeue { } __aligned(CACHE_LINE_SIZE); #ifndef VM_BATCHQUEUE_SIZE -#define VM_BATCHQUEUE_SIZE 7 +#define VM_BATCHQUEUE_SIZE 15 #endif struct vm_batchqueue { @@ -356,15 +356,17 @@ vm_batchqueue_init(struct vm_batchqueue *bq) bq->bq_cnt = 0; } -static inline bool +static inline int vm_batchqueue_insert(struct vm_batchqueue *bq, vm_page_t m) { + int slots_free; - if (bq->bq_cnt < nitems(bq->bq_pa)) { + slots_free = nitems(bq->bq_pa) - bq->bq_cnt; + if (slots_free > 0) { bq->bq_pa[bq->bq_cnt++] = m; - return (true); + return (slots_free); } - return (false); + return (slots_free); } static inline vm_page_t