From nobody Thu Aug 03 21:14:07 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RH1nM63NJz4V1b9 for ; Thu, 3 Aug 2023 21:14:31 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RH1nM2B7Tz3PxL; Thu, 3 Aug 2023 21:14:30 +0000 (UTC) (envelope-from eugen@grosbein.net) Authentication-Results: mx1.freebsd.org; none Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.17.1/8.17.1) with ESMTPS id 373LEQ6j049281 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Thu, 3 Aug 2023 21:14:27 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: asomers@freebsd.org Received: from [10.58.0.11] (dadvw [10.58.0.11] (may be forged)) by eg.sd.rdtc.ru (8.17.1/8.17.1) with ESMTPS id 373LEPlK005746 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 4 Aug 2023 04:14:25 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Best practices for cxgbei and link aggregation? To: Alan Somers , FreeBSD References: From: Eugene Grosbein Message-ID: <708494cd-9c88-974e-da5f-1e8c1f270950@grosbein.net> Date: Fri, 4 Aug 2023 04:14:07 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT autolearn=disabled version=3.4.6 X-Spam-Report: * -0.0 SHORTCIRCUIT No description available. * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on hz.grosbein.net X-Rspamd-Queue-Id: 4RH1nM2B7Tz3PxL X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE] 04.08.2023 0:56, Alan Somers wrote: > I'm trying to build a high-speed iSCSI server. I have two Chelsio T6 > cards providing 4x 25GbE ports. I have a requirement for > high-availability networking, and I also need multiple ports' worth of > bandwidth. What's the best way to use them? > > First I tried LACP, of course. That works. But it doesn't work in > combination with cxgbei iSCSI offload. The clients can't connect. > This makes sense, because the offload engine probably requires all > packets from a single iSCSI session to enter and leave through the > same network port. With LACP, that won't be the case. This is a common misunderstanding. In fact, LACP has nothing to do with per-port traffic distribution. Used kind of hashing function is not a part of LACP. You just need to use link partners capable of good hashing. For example, FreeBSD lagg(4) hashing function is capable of using L2 (MAC), L3 (IP) and L4 (TCP/UDP ports) headers and any kind of combination of such headers to make sure that packets if single flow go out using same port. Take a look at ifconfig.8 manual page for "lagghash".