From nobody Sun Mar 17 15:40:54 2024 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TyMfG51mlz5Dbxf; Sun, 17 Mar 2024 15:41:26 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TyMfG3Mynz4206; Sun, 17 Mar 2024 15:41:26 +0000 (UTC) (envelope-from gallatin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710690086; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DkBGEP9m7putnBRP0xqmnfpDf7kPZmyEVHCztyOZmIg=; b=KL9aF7nJcT5zv08irj9QfpDTgEKF9hm+p8ulksTHeVNAWl5Oj7JWWlrdXV84p/0ZhtQfzb C7suFsp9W8TYhUIY1HAnkUdiOQ7RpQ32F3nDGYm7MpjB8M+sK/fd0H1bgoZGciiqaJrEIw RvcKoR3fjqK4E511bV6dErC2gBmxEHZcZrPrWaT7xHXkx6GsvqqHPrxwEa2o6PZxIdG9OW lLmzGUoPEz0pP9R4L0AhgElS2jR3XUNV0OLfBW4ongRYnBDWbkWVogVFLcLtDEtqYR7Klh U/sHxGD+zMtsTh8TLbEMiqXLN/DNieBv3BjdSIpYMgl+OdSdxPab3ddIxVdfyw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1710690086; a=rsa-sha256; cv=none; b=oliMj5JRTZpKSNziQaOT74z0Uz4DzlE8xTcEKLXIsicU8W/p8NUtbXkSKz2kgBDqYHJauv f1Zij+s6DuVEN2gATv0oVUFbinqrZnEaII2M8mmtm1rRb3tfYmrS2GiNPU04M9uaWEVCh9 Z2COYA5ooj8zy4U5NksUlu4tvCWHA/GkYncrdn3MpYIvmtzVWVAAVHlsGvBLMI8heiuQNC EZXeuM8hdIdrllcuLIIf0ZD05XfLA0eX/+lgCP9/vOH9QdC5gA1pxZ6PKpNVV6T6qPMitD 8/hQVTJtUOujyqaWJpqOG+Osyf/wG7Qotl6yYMnocRa6rD6vZGmerM1gPyRVqQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1710690086; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=DkBGEP9m7putnBRP0xqmnfpDf7kPZmyEVHCztyOZmIg=; b=kBJa7HXFKEpEfXsefEuGdqfW2E3Hl2xAsSwcA6qxq0oiX17OcMjyFH9OBq0t5ExjBPMk77 J0r5j/zeucsDHLSAygkNxGp3924WxcotEEAhE37uKsHSyD5qGizpGxD7PG4I4v2IjoXtHX FlnKoOiUTVqMCxX1/sVfu9E8ro5UiezTrp5d2nJrrTUoCdvkWc+szjCopTK7U5YJMlDsD9 HduzLKV3rjzotahNbfN0Ek8J8udlAx657rpOi7BvBkx0uI6G4Byc5tYnXPJLpPN/vZPe5Z EpjiKlteL7cxPJXmuE8lyfFriJXFSHQ4YREc9OGZ5lSSgNjUcin4P9ZYtETwLQ== Received: from fauth2-smtp.messagingengine.com (fauth2-smtp.messagingengine.com [103.168.172.201]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: gallatin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4TyMfF3bFmz14mC; Sun, 17 Mar 2024 15:41:25 +0000 (UTC) (envelope-from gallatin@freebsd.org) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfauth.nyi.internal (Postfix) with ESMTP id 164481200043; Sun, 17 Mar 2024 11:41:25 -0400 (EDT) Received: from imap53 ([10.202.2.103]) by compute5.internal (MEProxy); Sun, 17 Mar 2024 11:41:25 -0400 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeehgdehiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepofgfggfkjghffffhvfevufgtsehmtd erreerredtnecuhfhrohhmpedfffhrvgifucfirghllhgrthhinhdfuceoghgrlhhlrght ihhnsehfrhgvvggsshgurdhorhhgqeenucggtffrrghtthgvrhhnpeetueeftdehgeekle ffjeehhfelgedtgffftdeuhfdtgeevudevledukedtffefffenucevlhhushhtvghrufhi iigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehgrghllhgrthhinhdomhgvshhmth hprghuthhhphgvrhhsohhnrghlihhthidqudeffeehledvvdduiedqvdelhedtgedukeeg qdhgrghllhgrthhinheppehfrhgvvggsshgurdhorhhgsehfrghsthhmrghilhdrtghomh X-ME-Proxy: Feedback-ID: i41414658:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id CAA0236400AB; Sun, 17 Mar 2024 11:41:24 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-300-gdee1775a43-fm-20240315.001-gdee1775a List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Message-Id: In-Reply-To: <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> References: <42C327BD-6CE4-43AA-A1AE-3BEC08D623DB@freebsd.org> <486915F0-456B-4B09-A8BC-93BBA79C4CA1@freebsd.org> <20240313080624.6c73908c@ernst.home> <508E3B47-8E1B-469F-97B1-2171A3098888@freebsd.org> <86a5n1i0xg.fsf@ltc.des.dev> <78D1FF09-71A3-4486-B934-D8332F54B237@freebsd.org> <20240316104053.20bef8c2@ernst.home> <20240316115128.33d11f7b@ernst.home> <7367F29A-D52B-4828-B79A-AA2667E81E7D@freebsd.org> <4FF534F6-B35D-4596-8D1E-226AD1347AC8@freebsd.org> <6e795e9c-8de4-4e02-9a96-8fabfaa4e66f@app.fastmail.com> Date: Sun, 17 Mar 2024 11:40:54 -0400 From: "Drew Gallatin" To: "Nuno Teixeira" , tuexen@freebsd.org Cc: garyj@gmx.de, current@freebsd.org, net@freebsd.org, "Randall Stewart" Subject: Re: Request for Testing: TCP RACK Content-Type: multipart/mixed; boundary=e58bff945182484499f23ee3bb676b2a --e58bff945182484499f23ee3bb676b2a Content-Type: multipart/alternative; boundary=43607613842b4113a49b4bd429aef6fd --43607613842b4113a49b4bd429aef6fd Content-Type: text/plain Resending with the patch as an attachment. Drew On Sun, Mar 17, 2024, at 11:39 AM, Drew Gallatin wrote: > I don't have the full context, but it seems like the complaint is a performance regression in bonnie++ and perhaps other things when tcp_hpts is loaded, even when it is not used. Is that correct? > > If so, I suspect its because we drive the tcp_hpts_softclock() routine from userret(), in order to avoid tons of timer interrupts and context switches. To test this theory, you could apply a patch like: > > diff --git a/sys/kern/subr_trap.c b/sys/kern/subr_trap.c > index e9a16cd0b36e..54b540c97123 100644 > --- a/sys/kern/subr_trap.c > +++ b/sys/kern/subr_trap.c > @@ -138,7 +138,7 @@ userret(struct thread *td, struct trapframe *frame) > * Software Timer Support for Network Processing" > * by Mohit Aron and Peter Druschel. > */ > - tcp_hpts_softclock(); > + /*tcp_hpts_softclock();*/ > /* > * Let the scheduler adjust our priority etc. > */ > > > If that fixes it, I suspect we should either make this hook optional for casual users of tcp_hpts(), or add some kind of "last called" timestamp to prevent it being called over and over and over on workloads which are syscall heavy. > > Note that for non-casual users of hpts (like Netflix, with hundreds of thousands of TCP connections managed by hpts), this call is a huge win, so I think we'd prefer that it remain in some form. > > Drew > --43607613842b4113a49b4bd429aef6fd Content-Type: text/html Content-Transfer-Encoding: quoted-printable
Resending = with the patch as an attachment.

Drew

On Sun, Mar 17, 2024, at 11:39 AM, Drew Gallatin= wrote:
I d= on't have the full context, but it seems like the complaint is a perform= ance regression in bonnie++ and perhaps other things when tcp_hpts is lo= aded, even when it is not used.  Is that correct?
If so, I suspect its because we drive the tcp_hpts_softclock= () routine from userret(), in order to avoid tons of timer interrupts an= d context switches.  To test this theory,  you could apply a p= atch like:

diff --git a/sys/kern/subr_trap.= c b/sys/kern/subr_trap.c
index e9a16cd0b36e..54b540c97123 = 100644
--- a/sys/kern/subr_trap.c
+++ b/sys/= kern/subr_trap.c
@@ -138,7 +138,7 @@ userret(struct thread= *td, struct trapframe *frame)
    &nb= sp;    * Software Timer Support for Network Processing"
         * by Mohit= Aron and Peter Druschel.
     &n= bsp;   */
-       = tcp_hpts_softclock();
+      = ; /*tcp_hpts_softclock();*/
     =    /*
       =   * Let the scheduler adjust our priority etc.
 =         */


If that fixes it, I suspect we should either make this= hook optional for casual users of tcp_hpts(), or add some kind of "last= called" timestamp to prevent it being called over and over and over on = workloads which are syscall heavy.

Note tha= t for non-casual users of hpts (like Netflix, with hundreds of thousands= of TCP connections managed by hpts), this call is a huge win, so I thin= k we'd prefer that it remain in some form.

= Drew

--43607613842b4113a49b4bd429aef6fd-- --e58bff945182484499f23ee3bb676b2a Content-Disposition: attachment; filename="nerf_tcp_hpts_softclock.diff" Content-Type: application/octet-stream; name="nerf_tcp_hpts_softclock.diff" Content-Transfer-Encoding: BASE64 ZGlmZiAtLWdpdCBhL3N5cy9rZXJuL3N1YnJfdHJhcC5jIGIvc3lzL2tlcm4vc3Vicl90cmFw LmMKaW5kZXggZTlhMTZjZDBiMzZlLi41NGI1NDBjOTcxMjMgMTAwNjQ0Ci0tLSBhL3N5cy9r ZXJuL3N1YnJfdHJhcC5jCisrKyBiL3N5cy9rZXJuL3N1YnJfdHJhcC5jCkBAIC0xMzgsNyAr MTM4LDcgQEAgdXNlcnJldChzdHJ1Y3QgdGhyZWFkICp0ZCwgc3RydWN0IHRyYXBmcmFtZSAq ZnJhbWUpCiAJICogU29mdHdhcmUgVGltZXIgU3VwcG9ydCBmb3IgTmV0d29yayBQcm9jZXNz aW5nIgogCSAqIGJ5IE1vaGl0IEFyb24gYW5kIFBldGVyIERydXNjaGVsLgogCSAqLwotCXRj cF9ocHRzX3NvZnRjbG9jaygpOworCS8qdGNwX2hwdHNfc29mdGNsb2NrKCk7Ki8KIAkvKgog CSAqIExldCB0aGUgc2NoZWR1bGVyIGFkanVzdCBvdXIgcHJpb3JpdHkgZXRjLgogCSAqLwo= --e58bff945182484499f23ee3bb676b2a--