From nobody Sat May 21 14:58:39 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 363F61AEB1F0; Sat, 21 May 2022 14:58:58 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-vk1-xa29.google.com (mail-vk1-xa29.google.com [IPv6:2607:f8b0:4864:20::a29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4L56Dd2qnkz4R2n; Sat, 21 May 2022 14:58:57 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: by mail-vk1-xa29.google.com with SMTP id bs5so5289016vkb.4; Sat, 21 May 2022 07:58:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=73LLaEzcRWLrA1cadyaxShn0oIQhPq4emWI4kh0NC8E=; b=KJbqSugoHwpdf/oW9/rhQMwZBMif9irahx2rvnJfXV1grveZj6qIBW6fytk2Sh3E1X nvObADaJROo71J5rtp7IgFBTMiNmXhjOS38FRtd5GirIJpV8EKHWJ/hpAjAfIHkDAv+I keUTObM5ObWm7dxg5IRKMjxYTDeFLMDvyOEDa5TBZgtP8lnSAiXP1P+UvENRzeQinopG UPbZL2V5b4mlh1CcUcjQqrA/EUjeeFikZuD6QPjh+ybb4jw06N0tS2fPv4gvQSJQKry6 90MFD7etUE1Znlvln65ia//D+jtexv6dHrP8JBx7cAfukCSPw5FktcVVioU19KnInTEL Dq5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=73LLaEzcRWLrA1cadyaxShn0oIQhPq4emWI4kh0NC8E=; b=ynVWf42/x7FZVBitd0lCl055nKQzlEOJ6hMoclnq3CZWidwh2XS5GK2UJRrK57BqHi FYE5+Q7tEDmavdJ22B1/9MlHin0yQwApZctr2aVYoH2eNmUBUnAbWRpodJB52Z9XDKAy 4gMNXDyIuGTg7bTDgSnCb1j4XUAl/ORVx6x5/1YIpzuQdOltch6IGNN+LP47VTDs2irn TEEsAtJ3JK4dEYl0pAc4caZ/RZQv28Jta1xsR+ihaKBv8XnAOFBIsNz9zyH2VnxiI3jd WI/prneWD1FYnRq5Y/1toA9dOO5gnpAwrqYGMQIC7K6lVDliQlhTINniHxAlxsJAExke FkTQ== X-Gm-Message-State: AOAM533LMEI26HIAmmciidVl78H4WE8sOnTbn6H1h9UlaLM7soGb3aUS amZhxM4kj1Alhht+XYj/96kYw2G3mYO74n8mneIoTDeEBW6wPw== X-Google-Smtp-Source: ABdhPJzCbV7gwDAxl0yTSTi/7s4w+D4OWYmr9idOG0F4zS65sCgLI83F242sqHw+V+7PXSsyoSwmtMTOeehcZ4Ss7Rg= X-Received: by 2002:a1f:ad0f:0:b0:352:3b:ce84 with SMTP id w15-20020a1fad0f000000b00352003bce84mr5589953vke.13.1653145130531; Sat, 21 May 2022 07:58:50 -0700 (PDT) List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 References: <202111161852.1AGIqhxc017891@gitrepo.freebsd.org> In-Reply-To: <202111161852.1AGIqhxc017891@gitrepo.freebsd.org> From: Sergey Kandaurov Date: Sat, 21 May 2022 18:58:39 +0400 Message-ID: Subject: Re: git: fcaa890c4469 - main - mbuf: Only allow extpg mbufs if the system has a direct map To: Mark Johnston Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Content-Type: multipart/alternative; boundary="0000000000004481c805df86d8e0" X-Rspamd-Queue-Id: 4L56Dd2qnkz4R2n X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=KJbqSugo; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of pluknet@gmail.com designates 2607:f8b0:4864:20::a29 as permitted sender) smtp.mailfrom=pluknet@gmail.com X-Spamd-Result: default: False [-3.66 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FREEFALL_USER(0.00)[pluknet]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::a29:from]; NEURAL_HAM_SHORT(-0.66)[-0.659]; MLMMJ_DEST(0.00)[dev-commits-src-all,dev-commits-src-main]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --0000000000004481c805df86d8e0 Content-Type: text/plain; charset="UTF-8" On Tue, 16 Nov 2021 at 22:52, Mark Johnston wrote: > The branch main has been updated by markj: > > URL: > https://cgit.FreeBSD.org/src/commit/?id=fcaa890c4469118255d463495b4044eef484fa3e > > commit fcaa890c4469118255d463495b4044eef484fa3e > Author: Mark Johnston > AuthorDate: 2021-11-16 18:31:04 +0000 > Commit: Mark Johnston > CommitDate: 2021-11-16 18:31:04 +0000 > > mbuf: Only allow extpg mbufs if the system has a direct map > > Some upcoming changes will modify software checksum routines like > in_cksum() to operate using m_apply(), which uses the direct map to > access packet data for unmapped mbufs. This approach of course does > not > work on platforms without a direct map, so we have to disallow the use > of unmapped mbufs on such platforms. > > I believe this is the right tradeoff: we only configure KTLS on amd64 > and arm64 today (and one KTLS consumer, NFS TLS, requires a direct map > already), and the use of unmapped mbufs with plain sendfile is a recent > optimization. If need be, m_apply() could be modified to create > CPU-private mappings of extpg mbuf pages as a fallback. > > So, change mb_use_ext_pgs to be hard-wired to zero on systems without a > direct map. Note that PMAP_HAS_DMAP is not a compile-time constant on > some systems, so the default value of mb_use_ext_pgs has to be > determined during boot. > > Reviewed by: jhb > Discussed with: gallatin > MFC after: 2 weeks > Sponsored by: The FreeBSD Foundation > Differential Revision: https://reviews.freebsd.org/D32940 > --- > sys/kern/kern_mbuf.c | 32 ++++++++++++++++++++++++++++++-- > sys/rpc/rpcsec_tls/rpctls_impl.c | 2 +- > 2 files changed, 31 insertions(+), 3 deletions(-) > > diff --git a/sys/kern/kern_mbuf.c b/sys/kern/kern_mbuf.c > index d1f2fd2bd9e4..78a270189a4b 100644 > --- a/sys/kern/kern_mbuf.c > +++ b/sys/kern/kern_mbuf.c > @@ -116,9 +116,26 @@ int nmbjumbop; /* limits number > of page size jumbo clusters */ > int nmbjumbo9; /* limits number of 9k jumbo clusters */ > int nmbjumbo16; /* limits number of 16k jumbo > clusters */ > > -bool mb_use_ext_pgs = true; /* use M_EXTPG mbufs for sendfile & TLS */ > -SYSCTL_BOOL(_kern_ipc, OID_AUTO, mb_use_ext_pgs, CTLFLAG_RWTUN, > +bool mb_use_ext_pgs = false; /* use M_EXTPG mbufs for sendfile & TLS */ > Hi, Does it mean that mb_use_ext_pgs has to be enabled manually now in head and releng/13.1 ? (it was on by default in releng/13.0) I failed to see how it still can be on by default from this change. What about initializing to true under #if PMAP_HAS_DMAP ? > > + > +static int > +sysctl_mb_use_ext_pgs(SYSCTL_HANDLER_ARGS) > +{ > + int error, extpg; > + > + extpg = mb_use_ext_pgs; > + error = sysctl_handle_int(oidp, &extpg, 0, req); > + if (error == 0 && req->newptr != NULL) { > + if (extpg != 0 && !PMAP_HAS_DMAP) > + error = EOPNOTSUPP; > + else > + mb_use_ext_pgs = extpg != 0; > + } > + return (error); > +} > +SYSCTL_PROC(_kern_ipc, OID_AUTO, mb_use_ext_pgs, CTLTYPE_INT | CTLFLAG_RW, > &mb_use_ext_pgs, 0, > + sysctl_mb_use_ext_pgs, "IU", > "Use unmapped mbufs for sendfile(2) and TLS offload"); > > static quad_t maxmbufmem; /* overall real memory limit for all mbufs > */ > @@ -137,6 +154,7 @@ static void > tunable_mbinit(void *dummy) > { > quad_t realmem; > + int extpg; > > /* > * The default limit for all mbuf related memory is 1/2 of all > @@ -173,6 +191,16 @@ tunable_mbinit(void *dummy) > if (nmbufs < nmbclusters + nmbjumbop + nmbjumbo9 + nmbjumbo16) > nmbufs = lmax(maxmbufmem / MSIZE / 5, > nmbclusters + nmbjumbop + nmbjumbo9 + nmbjumbo16); > + > + /* > + * Unmapped mbufs can only safely be used on platforms with a > direct > + * map. > + */ > + if (PMAP_HAS_DMAP) { > + extpg = mb_use_ext_pgs; > + TUNABLE_INT_FETCH("kern.ipc.mb_use_ext_pgs", &extpg); > + mb_use_ext_pgs = extpg != 0; > + } > } > SYSINIT(tunable_mbinit, SI_SUB_KMEM, SI_ORDER_MIDDLE, tunable_mbinit, > NULL); [..] --0000000000004481c805df86d8e0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Tue, 16 Nov 2021 at 22:52, Mark John= ston <markj@freebsd.org> wro= te:
The branch m= ain has been updated by markj:

URL: https://cgit.= FreeBSD.org/src/commit/?id=3Dfcaa890c4469118255d463495b4044eef484fa3e
commit fcaa890c4469118255d463495b4044eef484fa3e
Author:=C2=A0 =C2=A0 =C2=A0Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2021-11-16 18:31:04 +0000
Commit:=C2=A0 =C2=A0 =C2=A0Mark Johnston <markj@FreeBSD.org>
CommitDate: 2021-11-16 18:31:04 +0000

=C2=A0 =C2=A0 mbuf: Only allow extpg mbufs if the system has a direct map
=C2=A0 =C2=A0 Some upcoming changes will modify software checksum routines = like
=C2=A0 =C2=A0 in_cksum() to operate using m_apply(), which uses the direct = map to
=C2=A0 =C2=A0 access packet data for unmapped mbufs.=C2=A0 This approach of= course does not
=C2=A0 =C2=A0 work on platforms without a direct map, so we have to disallo= w the use
=C2=A0 =C2=A0 of unmapped mbufs on such platforms.

=C2=A0 =C2=A0 I believe this is the right tradeoff: we only configure KTLS = on amd64
=C2=A0 =C2=A0 and arm64 today (and one KTLS consumer, NFS TLS, requires a d= irect map
=C2=A0 =C2=A0 already), and the use of unmapped mbufs with plain sendfile i= s a recent
=C2=A0 =C2=A0 optimization.=C2=A0 If need be, m_apply() could be modified t= o create
=C2=A0 =C2=A0 CPU-private mappings of extpg mbuf pages as a fallback.

=C2=A0 =C2=A0 So, change mb_use_ext_pgs to be hard-wired to zero on systems= without a
=C2=A0 =C2=A0 direct map.=C2=A0 Note that PMAP_HAS_DMAP is not a compile-ti= me constant on
=C2=A0 =C2=A0 some systems, so the default value of mb_use_ext_pgs has to b= e
=C2=A0 =C2=A0 determined during boot.

=C2=A0 =C2=A0 Reviewed by:=C2=A0 =C2=A0 jhb
=C2=A0 =C2=A0 Discussed with: gallatin
=C2=A0 =C2=A0 MFC after:=C2=A0 =C2=A0 =C2=A0 2 weeks
=C2=A0 =C2=A0 Sponsored by:=C2=A0 =C2=A0The FreeBSD Foundation
=C2=A0 =C2=A0 Differential Revision:=C2=A0 https://reviews.freebsd= .org/D32940
---
=C2=A0sys/kern/kern_mbuf.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|= 32 ++++++++++++++++++++++++++++++--
=C2=A0sys/rpc/rpcsec_tls/rpctls_impl.c |=C2=A0 2 +-
=C2=A02 files changed, 31 insertions(+), 3 deletions(-)

diff --git a/sys/kern/kern_mbuf.c b/sys/kern/kern_mbuf.c
index d1f2fd2bd9e4..78a270189a4b 100644
--- a/sys/kern/kern_mbuf.c
+++ b/sys/kern/kern_mbuf.c
@@ -116,9 +116,26 @@ int nmbjumbop;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* limits number of page size jumbo c= lusters */
=C2=A0int nmbjumbo9;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0/* limits number of 9k jumbo clusters */
=C2=A0int nmbjumbo16;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* limits number of 16k jumbo clusters */
-bool mb_use_ext_pgs =3D true;=C2=A0 =C2=A0 /* use M_EXTPG mbufs for sendfi= le & TLS */
-SYSCTL_BOOL(_kern_ipc, OID_AUTO, mb_use_ext_pgs, CTLFLAG_RWTUN,
+bool mb_use_ext_pgs =3D false;=C2=A0 =C2=A0/* use M_EXTPG mbufs for sendfi= le & TLS */

Hi,
Does it mean that mb_use_ext_pgs has to be enabled manually now=
in head and releng/13.1 ? (it was on by default in releng/13.0)<= /div>
I failed to see how it still can be on by default from this chang= e.

What about initializing to tru= e under #if PMAP_HAS_DMAP ?
=C2=A0
+
+static int
+sysctl_mb_use_ext_pgs(SYSCTL_HANDLER_ARGS)
+{
+=C2=A0 =C2=A0 =C2=A0 =C2=A0int error, extpg;
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0extpg =3D mb_use_ext_pgs;
+=C2=A0 =C2=A0 =C2=A0 =C2=A0error =3D sysctl_handle_int(oidp, &extpg, 0= , req);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0if (error =3D=3D 0 && req->newptr != =3D NULL) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (extpg !=3D 0 &a= mp;& !PMAP_HAS_DMAP)
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0error =3D EOPNOTSUPP;
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0else
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0mb_use_ext_pgs =3D extpg !=3D 0;
+=C2=A0 =C2=A0 =C2=A0 =C2=A0}
+=C2=A0 =C2=A0 =C2=A0 =C2=A0return (error);
+}
+SYSCTL_PROC(_kern_ipc, OID_AUTO, mb_use_ext_pgs, CTLTYPE_INT | CTLFLAG_RW,=
=C2=A0 =C2=A0 =C2=A0&mb_use_ext_pgs, 0,
+=C2=A0 =C2=A0 sysctl_mb_use_ext_pgs, "IU",
=C2=A0 =C2=A0 =C2=A0"Use unmapped mbufs for sendfile(2) and TLS offloa= d");

=C2=A0static quad_t maxmbufmem;=C2=A0 =C2=A0 =C2=A0 /* overall real memory = limit for all mbufs */
@@ -137,6 +154,7 @@ static void
=C2=A0tunable_mbinit(void *dummy)
=C2=A0{
=C2=A0 =C2=A0 =C2=A0 =C2=A0 quad_t realmem;
+=C2=A0 =C2=A0 =C2=A0 =C2=A0int extpg;

=C2=A0 =C2=A0 =C2=A0 =C2=A0 /*
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* The default limit for all mbuf related = memory is 1/2 of all
@@ -173,6 +191,16 @@ tunable_mbinit(void *dummy)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (nmbufs < nmbclusters + nmbjumbop + nmbju= mbo9 + nmbjumbo16)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 nmbufs =3D lmax(max= mbufmem / MSIZE / 5,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 nmbcl= usters + nmbjumbop + nmbjumbo9 + nmbjumbo16);
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0/*
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 * Unmapped mbufs can only safely be used on pl= atforms with a direct
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 * map.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 */
+=C2=A0 =C2=A0 =C2=A0 =C2=A0if (PMAP_HAS_DMAP) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0extpg =3D mb_use_ex= t_pgs;
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0TUNABLE_INT_FETCH(&= quot;kern.ipc.mb_use_ext_pgs", &extpg);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mb_use_ext_pgs =3D = extpg !=3D 0;
+=C2=A0 =C2=A0 =C2=A0 =C2=A0}
=C2=A0}
=C2=A0SYSINIT(tunable_mbinit, SI_SUB_KMEM, SI_ORDER_MIDDLE, tunable_mbinit,= NULL);
=C2=A0[..]

--0000000000004481c805df86d8e0--