From nobody Thu Aug 03 17:56:28 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RGxP70Pphz4TlPD for ; Thu, 3 Aug 2023 17:56:43 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vs1-f50.google.com (mail-vs1-f50.google.com [209.85.217.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RGxP56pW2z4mDm for ; Thu, 3 Aug 2023 17:56:41 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.217.50 as permitted sender) smtp.mailfrom=asomers@gmail.com; dmarc=none Received: by mail-vs1-f50.google.com with SMTP id ada2fe7eead31-447abf05ea4so460274137.0 for ; Thu, 03 Aug 2023 10:56:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691085400; x=1691690200; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=IfMiC58zzcVnGE5mOMO12rGGRhp5axiOfGUNTecYcH8=; b=FXzK/UdDE9GvM4Rw80PN/i4MQPhct8PaRtRECu2jVFjtyFBuSWKzFeOYFhdBqY1ZA/ ThNvmNxqvfFxftfyiqU0Py6x5nv8hEGahIJMB74DebzxviV3UumChyvLfnC9Pei/n2Ur bRrBCrjHfDvr1BQJYcjFQQnfyBGuRpYmz7izPEolNpy7b1hcIL5694VGa1kXaTpqTIhU 98uvMWIUJqOwLpAOo8CbElUR3nEEZVRDzdIyBaBCABTVXUTUdaQJZndB913uKCr89o+s FhaiH9MFvLf35IpQfgXwdUdgE0UxmizJyQKdxXUsbdE/iH0lx7fNxBSpNDfYmUycIp/G DBQQ== X-Gm-Message-State: ABy/qLbhUG2CRdRxDvdblSc355sejDaPgqEt8ovg43HA1O+mCbyLGL1P ZapvkWslmC029GKNQbFVOeG+mgh6lPmX6UwK/378mgG+ X-Google-Smtp-Source: APBJJlFnc+XesptbRICVWBzhMnE80XBpYoU7qm8+k/OTspe354/qenrLM3Xjoqj50cuGjn+qH/t8UzsudXkckVMhAbo= X-Received: by 2002:a05:6102:410:b0:443:682e:2088 with SMTP id d16-20020a056102041000b00443682e2088mr5672591vsq.12.1691085400199; Thu, 03 Aug 2023 10:56:40 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 From: Alan Somers Date: Thu, 3 Aug 2023 10:56:28 -0700 Message-ID: Subject: Best practices for cxgbei and link aggregation? To: FreeBSD Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [1.99 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_SPAM_SHORT(0.99)[0.993]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.217.50:from]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[209.85.217.50:from]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[asomers]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DOM_EQ_FROM_DOM(0.00)[] X-Spamd-Bar: + X-Rspamd-Queue-Id: 4RGxP56pW2z4mDm I'm trying to build a high-speed iSCSI server. I have two Chelsio T6 cards providing 4x 25GbE ports. I have a requirement for high-availability networking, and I also need multiple ports' worth of bandwidth. What's the best way to use them? First I tried LACP, of course. That works. But it doesn't work in combination with cxgbei iSCSI offload. The clients can't connect. This makes sense, because the offload engine probably requires all packets from a single iSCSI session to enter and leave through the same network port. With LACP, that won't be the case. Next I tried ALUA. I disabled LACP and assigned a different IP address to each network port. On the clients I connected to each target via all four ports. Then I used gmultipath for each target in Active/Active mode. This did not work, even with cxgbei disabled. iSCSI requests were well-distributed, but the server sent all responses out of cc0. It seems that ctld doesn't do any kind of source-routing. Using cxgbei makes the situation even worse. Without source-routing, the clients almost immediately get disconnected. No doubt that's because requests and responses are going through different network ports again. Is there any way to use cxgbei iSCSI offload with multiple NICs? My best idea, which I haven't yet tried, is to assign each NIC to a different VLAN, then use ALUA over those. That should ensure that requests and responses for each iSCSI session go through the same port. However, gmultipath would still round-robin requests for each iSCSI target through different sessions. I don't know if that would be a problem. -Alan