From nobody Wed Oct 23 23:43:17 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XYlwx2nPQz5Znsf for ; Wed, 23 Oct 2024 23:43:29 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-yw1-x112d.google.com (mail-yw1-x112d.google.com [IPv6:2607:f8b0:4864:20::112d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XYlwx2Kbqz4mWd for ; Wed, 23 Oct 2024 23:43:29 +0000 (UTC) (envelope-from nparhar@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x112d.google.com with SMTP id 00721157ae682-6e34fa656a2so3487447b3.1 for ; Wed, 23 Oct 2024 16:43:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729727008; x=1730331808; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=PpBppLbdKKjpayb0pGphjeqtoA8xYV/HbeKEKBOoPZU=; b=aPYpWsu+4Xz7IW8jJJum7H/uqrJZH/pmjIFhKx5W2ciVsm0nFnvmgirYzGhQhGQWpY cYcUedretxxGMXJWquLMsTnhCr5OSN35wzEedj7pXvtnh6Y8/Ws05PdBeJLg5bfvjrvW L6Eq6n3NWBvWz0ZgZoD0DhsEV3ORF3QoECx+baNjQ74SXXSTY8W3v4dMRXGGF5HY7aeq 3Cwehku0jEagUUTbaDc302Nsv/J6Fj6QBkl5GEkOMGkDvP+ZkRMkqlLTEv4vwFON9QkR 4DfXdAUG9HtF2m87W6353JcPUfStr3kb3xauiVQwd2HcWwB1uxqBSiepfp2pxJH2JZyp 6H4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729727008; x=1730331808; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PpBppLbdKKjpayb0pGphjeqtoA8xYV/HbeKEKBOoPZU=; b=EXJiFvNkqlN8vaBRLJ8+Cpem9l7aVYAd8Y0PR92sl94Rv/kdrgbpJbQ1lVYkk2P9je BRDEEVPv3WfHQTsrNFdLtSKzGI6oILs419lvRiqEg/aI7qRewzLzIeSEz6VGtu3WZltA /mloCaCV22f1MWdcMbfxys/gF3nX2tKRD+x0LxKLB4Rbk1HNc//vBoj7l3u0qj4gEv55 l4/EUCwKI5QSHi/leXnF4pc5bF+Sbe/mQwI0q6Y+qPvaaV9+jELSqvSf/1O3rDF9C2Sc SWcoR1O0DGrBVHIoVCxDFaQiChBxKeRkz1h9TxFvBNHwycr1n9X+lG97p3fgMpqXpD9n ndtQ== X-Gm-Message-State: AOJu0YyXS9OgH0ntzKWpcET2v0x2JMUA48kIfzQJ0GqgOsoldQqRasi0 qri7mxI9I/RdZ2dR+E8sCUDIeyzFXvJvbuvy7+7Z4uoLRnNZK3q5hQwdHrXsDBgPx05BH18e+fu 9xBmmvhn0qFuJbMjPOx9YTeV6qPCfPugM X-Google-Smtp-Source: AGHT+IG5w1DWZeJyShPRsY4/82IQBEuJoRELXWMrT9+D3mO1Z9Xi6IeR0DeldgokNxlQ9uSmA5J7PUTQlIesLpjEL6U= X-Received: by 2002:a05:690c:f0d:b0:6e2:d2a:e998 with SMTP id 00721157ae682-6e7f0dc197fmr41837077b3.2.1729727008579; Wed, 23 Oct 2024 16:43:28 -0700 (PDT) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Navdeep Parhar Date: Wed, 23 Oct 2024 16:43:17 -0700 Message-ID: Subject: Re: Chelsio / cxlv(4): strange messages, SR-IOV interface does not work To: Lexi Winter Cc: freebsd-net@freebsd.org Content-Type: multipart/alternative; boundary="000000000000e7a81406252d733f" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4XYlwx2Kbqz4mWd X-Spamd-Bar: ---- --000000000000e7a81406252d733f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 22, 2024 at 11:21=E2=80=AFPM Lexi Winter wrot= e: > hello, > > i'm trying to configure a cxlv(4) device, which is a VF of a Chelsio > T540-CR on a host running bhyve. > > host: FreeBSD 15.0-CURRENT #3 lf/main-n269068-2cff93ced1d: Wed Oct 23 > 02:48:20 BST 2024 > guest: FreeBSD 15.0-CURRENT #2 lf/main-n269067-56dd459904b: Sat Oct 19 > 18:36:40 BST 2024 > > the VF appears correctly in the VM: > > root@lily:~ # kldload if_cxlv > t5vf0: mem > 0xc000e000-0xc000efff,0xc0000000-0xc0007fff,0xc0008000-0xc0009fff at > device 6.0 on pci0 > t5vf0: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq > cxlv0: on t5vf0 > cxlv0: 2 txq, 1 rxq (NIC) > > and after bringing the interface 'up' everything seems fine: > > root@lily:~ # ifconfig cxlv0 > cxlv0: flags=3D1008843 > metric 0 mtu 1500 > > > options=3D6ec07bb > ether 06:44:3f:e7:60:30 > media: Ethernet 10Gbase-Twinax (10Gbase-Twinax > ) > status: active > nd6 options=3D29 > > however, trying to assign an IP address causes immediate problems: > > root@lily:~ # ifconfig cxlv0 inet6 2001:8b0:aab5:7::10/64 > root@lily:~ # Oct 23 06:16:07 lily kernel: cxlv0: a looped back NS > message is detected during DAD for fe80:3::444:3fff:fee7:6030. Another > DAD probes are being sent. > root@lily:~ # dmesg|grep loop > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > cxlv0: a looped back NS message is detected during DAD for > fe80:3::444:3fff:fee7:6030. Another DAD probes are being sent. > > i find this strange because the link local IP address in the kernel > error is not even configured on the interface: > > root@lily:~ # ifconfig cxlv0 > cxlv0: flags=3D1008843 > metric 0 mtu 1500 > > > options=3D6ec07bb > ether 06:44:3f:e7:60:30 > inet6 2001:8b0:aab5:7::10/64 > inet6 fe80::444:3fff:fee7:6030%cxlv0/64 tentative scopeid 0x3 > media: Ethernet 10Gbase-Twinax (10Gbase-Twinax > ) > status: active > nd6 options=3D21 > > am i doing something wrong here? > You can disable IPv6 DAD as a workaround for this issue. The problem is that the VF's multicast tx is getting reflected back to it by the internal switch when it shouldn't. # sysctl net.inet6.ip6.dad_count=3D0 Regards, Navdeep > > thanks, lexi. > > --000000000000e7a81406252d733f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Tue, Oct 22, 2024 at 11:21=E2=80=AFPM Lexi Winter <lexi@le-fay.org> wrote:
hello,

i'm trying to configure a cxlv(4) device, which is a VF of a Chelsio T540-CR on a host running bhyve.

host: FreeBSD 15.0-CURRENT #3 lf/main-n269068-2cff93ced1d: Wed Oct 23
02:48:20 BST 2024
guest: FreeBSD 15.0-CURRENT #2 lf/main-n269067-56dd459904b: Sat Oct 19
18:36:40 BST 2024

the VF appears correctly in the VM:

root@lily:~ # kldload if_cxlv
t5vf0: <Chelsio T540-CR VF> mem
0xc000e000-0xc000efff,0xc0000000-0xc0007fff,0xc0008000-0xc0009fff at
device 6.0 on pci0
t5vf0: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq
cxlv0: <port 0> on t5vf0
cxlv0: 2 txq, 1 rxq (NIC)

and after bringing the interface 'up' everything seems fine:

root@lily:~ # ifconfig cxlv0
cxlv0: flags=3D1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP&g= t;
metric 0 mtu 1500

options=3D6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_H= WCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HW= RXTSTMP,MEXTPG>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ether 06:44:3f:e7:60:30
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0media: Ethernet 10Gbase-Twinax <full-d= uplex> (10Gbase-Twinax
<full-duplex,rxpause,txpause>)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0status: active
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0nd6 options=3D29<PERFORMNUD,IFDISABLED= ,AUTO_LINKLOCAL>

however, trying to assign an IP address causes immediate problems:

root@lily:~ # ifconfig cxlv0 inet6 2001:8b0:aab5:7::10/64
root@lily:~ # Oct 23 06:16:07 lily kernel: cxlv0: a looped back NS
message is detected during DAD for fe80:3::444:3fff:fee7:6030.=C2=A0 Anothe= r
DAD probes are being sent.
root@lily:~ # dmesg|grep loop
cxlv0: a looped back NS message is detected during DAD for
fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.
cxlv0: a looped back NS message is detected during DAD for
fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.
cxlv0: a looped back NS message is detected during DAD for
fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.
cxlv0: a looped back NS message is detected during DAD for
fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.
cxlv0: a looped back NS message is detected during DAD for
fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.
cxlv0: a looped back NS message is detected during DAD for
fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.
cxlv0: a looped back NS message is detected during DAD for
fe80:3::444:3fff:fee7:6030.=C2=A0 Another DAD probes are being sent.

i find this strange because the link local IP address in the kernel
error is not even configured on the interface:

root@lily:~ # ifconfig cxlv0
cxlv0: flags=3D1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP&g= t;
metric 0 mtu 1500

options=3D6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_H= WCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HW= RXTSTMP,MEXTPG>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ether 06:44:3f:e7:60:30
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0inet6 2001:8b0:aab5:7::10/64
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0inet6 fe80::444:3fff:fee7:6030%cxlv0/64 t= entative scopeid 0x3
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0media: Ethernet 10Gbase-Twinax <full-d= uplex> (10Gbase-Twinax
<full-duplex,rxpause,txpause>)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0status: active
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0nd6 options=3D21<PERFORMNUD,AUTO_LINKL= OCAL>

am i doing something wrong here?

You ca= n disable IPv6 DAD as a=20 workaround for this issue.=C2=A0 The problem is that the VF's multicast= tx is getting reflected back to it by the internal switch when it shouldn't.=

# sysctl net.inet6.ip6.dad_count=3D0

Regards,
Navdeep
=C2=A0

=C2=A0 =C2=A0 =C2=A0 =C2=A0 thanks, lexi.

--000000000000e7a81406252d733f--