From nobody Thu Aug 03 23:27:37 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RH4lN6JbnzZMkw for ; Thu, 3 Aug 2023 23:28:00 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RH4lN37YMz4KCc; Thu, 3 Aug 2023 23:28:00 +0000 (UTC) (envelope-from eugen@grosbein.net) Authentication-Results: mx1.freebsd.org; none Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.17.1/8.17.1) with ESMTPS id 373NRtW7050375 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Thu, 3 Aug 2023 23:27:56 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: asomers@freebsd.org Received: from [10.58.0.11] (dadvw [10.58.0.11] (may be forged)) by eg.sd.rdtc.ru (8.17.1/8.17.1) with ESMTPS id 373NRtKO006488 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 4 Aug 2023 06:27:55 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Best practices for cxgbei and link aggregation? To: Alan Somers References: <708494cd-9c88-974e-da5f-1e8c1f270950@grosbein.net> Cc: FreeBSD From: Eugene Grosbein Message-ID: <4b9141b2-3d0d-955f-e11c-ff5291543cdd@grosbein.net> Date: Fri, 4 Aug 2023 06:27:37 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT autolearn=disabled version=3.4.6 X-Spam-Report: * -0.0 SHORTCIRCUIT No description available. * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on hz.grosbein.net X-Rspamd-Queue-Id: 4RH4lN37YMz4KCc X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE] 04.08.2023 4:41, Alan Somers wrote: > On Thu, Aug 3, 2023 at 2:14 PM Eugene Grosbein wrote: >> >> 04.08.2023 0:56, Alan Somers wrote: >> >>> I'm trying to build a high-speed iSCSI server. I have two Chelsio T6 >>> cards providing 4x 25GbE ports. I have a requirement for >>> high-availability networking, and I also need multiple ports' worth of >>> bandwidth. What's the best way to use them? >>> >>> First I tried LACP, of course. That works. But it doesn't work in >>> combination with cxgbei iSCSI offload. The clients can't connect. >>> This makes sense, because the offload engine probably requires all >>> packets from a single iSCSI session to enter and leave through the >>> same network port. With LACP, that won't be the case. >> >> This is a common misunderstanding. In fact, LACP has nothing to do with per-port >> traffic distribution. Used kind of hashing function is not a part of LACP. >> You just need to use link partners capable of good hashing. >> >> For example, FreeBSD lagg(4) hashing function is capable of using L2 (MAC), L3 (IP) and L4 (TCP/UDP ports) >> headers and any kind of combination of such headers to make sure that packets if single flow go out >> using same port. Take a look at ifconfig.8 manual page for "lagghash". > > For iSCSI, any combination of those hash settings should result in a > flow that is always tied to a single port. But for cxgbei to work, I > need to ensure that the switch also hashes flows in the same way, to > the _exact_same_port_. Even if I had access to the switch's LACP > configuration, which I do not (but perhaps could acquire), is there > any guarantee that it would hash things the same way as FreeBSD? No. > Is the LACP hashing algorithm standardized? Hashing is not standardized. And hashing is not part of LACP, at all. LACP is not about hashing.