From nobody Sat Oct 12 00:20:22 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XQPKG0SQ4z5YPvH for ; Sat, 12 Oct 2024 00:20:34 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XQPKF5csnz4j1b for ; Sat, 12 Oct 2024 00:20:33 +0000 (UTC) (envelope-from nparhar@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-6e1f48e7c18so21936567b3.3 for ; Fri, 11 Oct 2024 17:20:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728692433; x=1729297233; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=dEVE8/s2STswuNForbaoP76GK1gxnpV5t53rnoASHIs=; b=UkGw96SiQ9slxgH0Q/UjIrktmh6W+pWO3UR6/8qBfh/Sg49q57MX3zc2ylTOx76ZFw l8c3yoWI8EobpHH3sSqFfRVRHFnDJJ/sAQryEVsLpf26zePxIPMmNDlnh6afWcF6ulh9 qinHkZZ3NzVPNxJDVg8LnQuahz76K4zYSBBjRqEECC8nN1w9K493NWyTwVM/g6q1A52h SzqqukL7bfTC2b3/B3mNbRe0IHo7xn7SCb27rpTClr81cR6RcFhpq6PPfFFLPFppCPyx xUAfmn2w65ssvsZ15NdK0Y3o5WvqGhfhb0nLALtcw0XhPICQpus1NTXZA9r+B2mnJnDZ 6uBQ== X-Gm-Message-State: AOJu0YwAw8tT1OdZOr9FjnNl3bBjG7AgVkXtXY3i0T+ibPD1QhZ6fCFI 3JMPwLHraZ/goc/igFu6MFrsci2KJDwtFxjxXH8MFj2Zrdsf5CeZel4gXBsvXAHTkD6l4XcwL1Y OJjmReMUB041hSK1UTx0b2uOgYOjs63Gs X-Google-Smtp-Source: AGHT+IH1yoq/T72zLXhJs9TM2CLXTUb8ymz8FMCMiO/1NbwvmCCm4ky2fovHMiItsC9/A+/wYnC2gDX7ZaKSCAvfYIA= X-Received: by 2002:a05:690c:c09:b0:6db:2604:ea6b with SMTP id 00721157ae682-6e3479dbd00mr37031877b3.25.1728692432787; Fri, 11 Oct 2024 17:20:32 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: <6BC00399-9D4C-4893-9FE4-4F99BD295909@jnielsen.net> In-Reply-To: <6BC00399-9D4C-4893-9FE4-4F99BD295909@jnielsen.net> From: Navdeep Parhar Date: Fri, 11 Oct 2024 17:20:22 -0700 Message-ID: Subject: Re: Traffic between cxgbe VFs and/or PF on a host To: John Nielsen Cc: freebsd-net@freebsd.org Content-Type: multipart/alternative; boundary="00000000000061e4cb06243c922f" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Queue-Id: 4XQPKF5csnz4j1b X-Spamd-Bar: ---- --00000000000061e4cb06243c922f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Oct 11, 2024 at 3:56=E2=80=AFPM John Nielsen w= rote: > Hi- > > I=E2=80=99m running a FreeBSD 14-STABLE host with a Chelstio T520. I have= a bhyve > VM (also running 14-STABLE) to which I have assigned a VF of the NIC. Tha= t > is all working as expected; the host can pass traffic using the PF cxl0 a= nd > the guest can pass traffic using the VF cxlv0. However the host cannot > communicate with the guest. I am looking in to the possibility of enablin= g > 802.1qbg / VEPA / reflective relay on the switch port but I=E2=80=99d lik= e to know > if the T5 can do that switching itself without sending the packets over t= he > wire. The marketing material says the card "integrates a high performance > packet switch=E2=80=9D but I don=E2=80=99t know how to configure that fun= ctionality on > FreeBSD or if this use case is supported. Can anyone shed some light on > that? > The PF driver's tx bypasses the internal switch by default and is not visible to the VFs because of that. Set this knob to force it go through the switch. hw.cxgbe.tx_vm_wr Setting this to 1 instructs the driver to use VM work requests to transmit data. This lets PF interfaces transmit frames to VF interfaces over the internal switch in the ASIC. Note that the cxgbev(4) VF driver always uses VM work requests and is not affected by this tunable. The default value is 0 and should be changed only if PF and VF interfaces need to communicate with each other. Different interfaces can be assigned different values using the dev..X.tx_vm_wr sysctl when the interface is administratively down. Regards, Navdeep > The other alternative would be to wire up the second port but if I can ge= t > away with not needing to use another SFP+ port on the switch for this tha= t > would be ideal. > > Thanks! > > JN > > > --00000000000061e4cb06243c922f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Fri, Oct 11, 2024 at 3:56=E2=80=AFPM J= ohn Nielsen <lists@jnielsen.net> wrote:
Hi-

I=E2=80=99m running a FreeBSD 14-STABLE host with a Chelstio T520. I have a= bhyve VM (also running 14-STABLE) to which I have assigned a VF of the NIC= . That is all working as expected; the host can pass traffic using the PF c= xl0 and the guest can pass traffic using the VF cxlv0. However the host can= not communicate with the guest. I am looking in to the possibility of enabl= ing 802.1qbg / VEPA / reflective relay on the switch port but I=E2=80=99d l= ike to know if the T5 can do that switching itself without sending the pack= ets over the wire. The marketing material says the card "integrates a = high performance packet switch=E2=80=9D but I don=E2=80=99t know how to con= figure that functionality on FreeBSD or if this use case is supported. Can = anyone shed some light on that?


=C2=A0 =C2=A0=C2=A0 hw.cxgbe.tx_vm_wr<= br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Setting this to 1 instru= cts the driver to use VM work requests to transmit data.
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0This lets PF interfaces transmit frames t= o VF interfaces over the internal switch in
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0the ASIC.=C2=A0 Note that the cxgbev(4) VF driver alway= s uses VM work requests and is not
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0affected by this tunable.=C2=A0 The default value is 0 and sho= uld be changed only if PF
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0and VF interfaces need to communicate with each other.=C2=A0 Different i= nterfaces can be
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0assigne= d different values using the dev.<port>.X.tx_vm_wr sysctl when the in= terface
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0is administrativ= ely down.

--00000000000061e4cb06243c922f--