From nobody Fri Sep 13 07:36:59 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X4mNH339Yz5WF1P for ; Fri, 13 Sep 2024 07:37:03 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X4mNG2bl4z4TwQ for ; Fri, 13 Sep 2024 07:37:02 +0000 (UTC) (envelope-from cryintothebluesky@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=MbAbyEY7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of cryintothebluesky@gmail.com designates 2a00:1450:4864:20::333 as permitted sender) smtp.mailfrom=cryintothebluesky@gmail.com Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-42bbffe38e6so14703305e9.0 for ; Fri, 13 Sep 2024 00:37:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726213021; x=1726817821; darn=freebsd.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:from:date:from:to:cc:subject:date:message-id :reply-to; bh=ei3eRXsR0qj1xn/aQka2w/a24Pl/29nD++RvXItFSVE=; b=MbAbyEY7svsHzEJFisVbxzUOxcYEMaPpBcuhmP8hLvXDwn4h+Ve3Lhk4Q7QfewPqbG TJoi2ym0Iv0lm2DG3YHwUdFmQpU8mwcokTDnyCEBheoPbM7qaNqOLSYkOHMtf+gMDC7l +XdbAcvpoETRgyjbMPVcDboYZ4FMeQAz9q3FapTR2C8kq2Cg7oDfNTtFbme+kLamjmaY 0aWMmGcx9Evt9jb04CxcE1Kt02Rt4MvpvjyfC4QyU2nEeY2kbMA3A2oK7Zzaw3vEVjf2 zsnlmpCTpmprpOtZiUIiRMm/hzcMAsuLpzmeOvj+O7ltaxlJFmK/Ro+jk6SWPDpaur+P tAhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726213021; x=1726817821; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ei3eRXsR0qj1xn/aQka2w/a24Pl/29nD++RvXItFSVE=; b=c3XOdeHMLXDKRQKEdHhLyRB7ewYeik0UnAy/MtOzNjWVdUf+oxYwTdOo0X+0Yts97I +6JFHWga6aDWQsfGiFb/IOS2BDhIOHomGQBnWzWuBsmWGVrxV7w1Cp00jPDL2eAfFJ6g przZQk601P2B6cEZBwBzi2jwG/ngqikQTwk56zG675Rw9F6m0HxG8pEh4kQ4srRn8Wjj +2EhOHp2fpekYE6KbV4TT8j6F7NIFrXW24NDvhwClmqBonvCOMX9JAwsfmBtWFGi78gg 4fjKRAIfqjo3cNt9sRe/UYEIxjJ8JTLvB+iwSB4+8Nm30ET10bLsxSW1XKzgnFGfPHtr e6Uw== X-Forwarded-Encrypted: i=1; AJvYcCXxU6IDLHlu50egpHkwA9GRMfhr2XWoX2ygga9beia2RDR6PP4BEcSdVkwYdm5+oNjQl0wuUsp1g0zdRA==@freebsd.org X-Gm-Message-State: AOJu0YxRZBA4bxEgH0C9buhJbfn2zcOM3RasT/z3EocaKV9bOdQDKEjk VPmIAOrLsL7NrcdN7MJxyMcUiAzqUwxO0gGRPVt4Qsluf5D3RVtoPS3b+w== X-Google-Smtp-Source: AGHT+IEDCrnd+YItga+5SAWqUu/RhfEEY30fC+n31KZX18xV1TzLhN3GJ1YHkdBMM1bCRUsI5tKNxQ== X-Received: by 2002:a05:600c:474c:b0:42b:ac80:52ea with SMTP id 5b1f17b1804b1-42cdb5097fcmr45022545e9.6.1726213020479; Fri, 13 Sep 2024 00:37:00 -0700 (PDT) Received: from z600.home.lan (29.129.159.143.dyn.plus.net. [143.159.129.29]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-378956ddcdcsm15994327f8f.100.2024.09.13.00.37.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2024 00:37:00 -0700 (PDT) Date: Fri, 13 Sep 2024 08:36:59 +0100 From: Sad Clouds Cc: Paul Procacci , freebsd-net@freebsd.org Subject: Re: Performance issues with vnet jails + epair + bridge Message-Id: <20240913083659.443548a87559c3cbaba4e9d8@gmail.com> In-Reply-To: <20240913080356.98ea2c352595ae0bbd9f0ce8@gmail.com> References: <20240912181618.7895d10ad5ff2ebae9883192@gmail.com> <20240913080356.98ea2c352595ae0bbd9f0ce8@gmail.com> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Bar: - X-Spamd-Result: default: False [-1.17 / 15.00]; MISSING_TO(2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.67)[-0.674]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::333:from] X-Rspamd-Queue-Id: 4X4mNG2bl4z4TwQ On Fri, 13 Sep 2024 08:03:56 +0100 Sad Clouds wrote: > I built new kernel with "options RSS" however TCP throughput performance > now decreased from 128 MiB/sec to 106 MiB/sec. > > Looks like the problem has shifted from epair to netisr > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 12 root -56 - 0B 272K CPU3 3 3:45 100.00% intr{swi1: netisr 0} > 11 root 187 ki31 0B 64K RUN 0 9:00 62.41% idle{idle: cpu0} > 11 root 187 ki31 0B 64K CPU2 2 9:36 61.23% idle{idle: cpu2} > 11 root 187 ki31 0B 64K RUN 1 8:24 55.03% idle{idle: cpu1} > 0 root -64 - 0B 656K - 2 0:50 21.50% kernel{epair_task_2} I think the issue may be to do with the genet driver itself. I think the hardware is limited to one CPU per send or receive interrupt. On Linux the best I can do is set SMP affinity for send on CPU0 and for receive on CPU1, but that still leaves 2 other CPUs idle. $ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 ... 37: 74141 0 0 0 GICv2 189 Level eth0 38: 43174 0 0 0 GICv2 190 Level eth0