From nobody Fri Jan 03 23:59:42 2025 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YQ0ts3Hyzz5kcDb for ; Sat, 04 Jan 2025 00:00:05 +0000 (UTC) (envelope-from lists@jnielsen.net) Received: from webmail5.jnielsen.net (webmail5.jnielsen.net [IPv6:2607:f170:34:11::b0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.freebsdsolutions.net", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YQ0tq5SH5z4S0F; Sat, 4 Jan 2025 00:00:03 +0000 (UTC) (envelope-from lists@jnielsen.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of lists@jnielsen.net designates 2607:f170:34:11::b0 as permitted sender) smtp.mailfrom=lists@jnielsen.net; dmarc=none Received: from smtpclient.apple ([IPv6:2607:fa18:e2fe:2d6a:f83a:ef89:b83d:49fa]) (authenticated bits=0) by webmail5.jnielsen.net (8.17.2/8.17.2) with ESMTPSA id 503NxrjT019808 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 3 Jan 2025 16:59:56 -0700 (MST) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail5.jnielsen.net: Host [IPv6:2607:fa18:e2fe:2d6a:f83a:ef89:b83d:49fa] claimed to be smtpclient.apple From: John Nielsen Content-Type: multipart/alternative; boundary="Apple-Mail=_A9F64ED3-7010-43A7-867D-F253F3E3B109" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3776.700.51.11.1\)) Subject: cxgbe VF vs multicast Message-Id: <4ED3FE1D-776C-4CE2-A676-975CDB757694@jnielsen.net> Date: Fri, 3 Jan 2025 16:59:42 -0700 Cc: Navdeep Parhar To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.3776.700.51.11.1) X-Rspamd-Queue-Id: 4YQ0tq5SH5z4S0F X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.79 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.989]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:6364, ipnet:2607:f170:30::/44, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCPT_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[jnielsen.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCVD_TLS_ALL(0.00)[] --Apple-Mail=_A9F64ED3-7010-43A7-867D-F253F3E3B109 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi- I just moved my router VM from a bridge + virtio setup to using a VF = (virtual function) on my Chelsio T520-LL-CR to improve performance. = It=E2=80=99s working beautifully for the most part, but I have noticed = two things that aren=E2=80=99t, both related to multicast. First, when using dhcpcd to assign IPv6 addresses to my LAN VLAN = interfaces, I get this spammed repeatedly on the console for both VLANs = where I use prefix delegation: kernel: cxlv0.2: a looped back NS message is detected during DAD for = [address]. Another DAD probes are being sent I was able to work around that (I think?) by setting = "net.inet6.ip6.dad_count=3D0=E2=80=9D in /etc/sysctl.conf (as hinted in = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D270859 talking about = an mlx5 VF). Second, CARP no longer works. At all. I turned off the peer completely = but the interfaces never came out of the BACKUP state. For now I have = removed CARP from the setup. Both IPv6 address assignment and CARP were working correctly in the = previous setup. Is the VF receiving its own multicast packets back from the PF or = virtual switch in the T5? That would explain both issues. In any case, = is there anything I can do about it? Both host and guest are FreeBSD 14-stable. In addition to the T5 VF the = host passes through an em interface to the guest entirely. Config excerpts from both machines below. Any help/insight appreciated. = Thanks! -John On the host I have: #/boot/loader.conf vmm_load=3D"YES" hw.vmm.amdvi.enable=3D"1" pptdevs=3D"14/0/0=E2=80=9D #em0 hw.cxgbe.tx_vm_wr=3D=E2=80=9C1=E2=80=9D #allow PF to talk to VF #/etc/rc.conf ifconfig_cxl0=3D"up" vlans_cxl0=3D"2 104" ifconfig_cxl0_2=3D"inet ${ADDR2}/24" ifconfig_cxl0_2_ipv6=3D"inet6 accept_rtadv auto_linklocal" ifconfig_cxl0_104=3D"inet ${ADDR104}/24" iovctl_files=3D"/etc/iov/cxl0.conf" #/etc/iov/cxl0.conf=20 PF { device : cxl0 num_vfs : 1 } VF-0 { passthrough : true; } On the guest I have: #/boot/loader.conf if_cxgbev_load=3D"YES" net.inet.ip.fw.default_to_accept=3D=E2=80=9C1=E2=80=9D #allow dhcp6 = before ipfw ruleset is loaded #/etc/rc.conf ifconfig_cxlv0=3D"up" vlans_cxlv0=3D"2 98 103" ifconfig_cxlv0_2=3D"inet ${MY_IP}/24" ifconfig_cxlv0_2_ipv6=3D"inet6 auto_linklocal" # set from dhcpcd via = prefix delegation #ifconfig_cxlv0_2_alias0=3D"inet vhid 1 advskew ${MY_SKEW} pass ${PASS} = alias 192.168.2.1/32" ifconfig_cxlv0_2_alias0=3D"inet alias 192.168.2.1/32" #ifconfig_cxlv0_103=3D"inet vhid 2 advskew ${MY_SKEW} pass ${PASS} = 192.168.103.1/24" ifconfig_cxlv0_103=3D"inet 192.168.103.1/24" #ifconfig_cxlv0_98=3D"inet vhid 3 advskew ${MY_SKEW} pass ${PASS} = 192.168.98.1/24" ifconfig_cxlv0_98=3D"inet 192.168.98.1/24" ifconfig_cxlv0_98_ipv6=3D"inet6 auto_linklocal" rtadvd_interfaces=3D"cxlv0.2 cxlv0.98" dhclient_program=3D"/usr/local/sbin/dhcpcd" #/etc/sysctl.conf net.inet.carp.preempt=3D1 net.inet6.ip6.dad_count=3D0 #/usr/local/etc/dhcpcd.conf duid persistent vendorclassid option classless_static_routes option interface_mtu option rapid_commit require dhcp_server_identifier slaac hwaddr allowinterfaces em0 waitip 4 waitip 6 nodelay timeout 10 interface em0 ia_na ia_pd 1/::/64 cxlv0.2/0/64/1 ia_pd 2/::/64 cxlv0.98/0/64/1 --Apple-Mail=_A9F64ED3-7010-43A7-867D-F253F3E3B109 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi-

I just moved my router VM = from a bridge + virtio setup to using a VF (virtual function) on my = Chelsio T520-LL-CR to improve performance. It=E2=80=99s working = beautifully for the most part, but I have noticed two things that = aren=E2=80=99t, both related to = multicast.

First, when using dhcpcd to assign = IPv6 addresses to my LAN VLAN interfaces, I get this spammed repeatedly = on the console for both VLANs where I use prefix = delegation:
kernel: cxlv0.2: a looped back NS message is = detected during DAD for [address].  Another DAD probes are being = sent

I was able to work around that (I think?) = by setting "net.inet6.ip6.dad_count=3D0=E2=80=9D in /etc/sysctl.conf (as = hinted in https:= //bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D270859 talking = about an mlx5 VF).

Second, CARP no longer = works. At all. I turned off the peer completely but the interfaces never = came out of the BACKUP state. For now I have removed CARP from the = setup.

Both IPv6 address assignment and CARP = were working correctly in the previous = setup.

Is the VF receiving its own multicast = packets back from the PF or virtual switch in the T5? That would explain = both issues. In any case, is there anything I can do about = it?

Both host and guest are FreeBSD 14-stable. = In addition to the T5 VF the host passes through an em interface to the = guest entirely.

Config excerpts from both = machines below. Any help/insight appreciated. = Thanks!

-John


=
On the host I = have:

#/boot/loader.conf
vmm_load=3D= "YES"
hw.vmm.amdvi.enable=3D"1"
pptdevs=3D"14/0/0=E2=80= =9D #em0
hw.cxgbe.tx_vm_wr=3D=E2=80=9C1=E2=80=9D #allow PF to = talk to = VF

#/etc/rc.conf
ifconfig_cx= l0=3D"up"
vlans_cxl0=3D"2 = 104"
ifconfig_cxl0_2=3D"inet = ${ADDR2}/24"
ifconfig_cxl0_2_ipv6=3D"inet6 accept_rtadv = auto_linklocal"
ifconfig_cxl0_104=3D"inet = ${ADDR104}/24"
iovctl_files=3D"/etc/iov/cxl0.conf"

#/etc/iov/cxl0.conf 
PF = {
       device : cxl0
  =      num_vfs : 1
}
VF-0 = {
        passthrough : = true;
}


On the = guest I = have:

#/boot/loader.conf
if_cxgbev= _load=3D"YES"
net.inet.ip.fw.default_to_accept=3D=E2=80=9C1=E2=80= =9D #allow dhcp6 before ipfw ruleset is = loaded

#/etc/rc.conf
ifconfi= g_cxlv0=3D"up"
vlans_cxlv0=3D"2 98 = 103"
ifconfig_cxlv0_2=3D"inet = ${MY_IP}/24"
ifconfig_cxlv0_2_ipv6=3D"inet6 auto_linklocal" # = set from dhcpcd via prefix = delegation
#ifconfig_cxlv0_2_alias0=3D"inet vhid 1 advskew = ${MY_SKEW} pass ${PASS} alias = 192.168.2.1/32"
ifconfig_cxlv0_2_alias0=3D"inet alias = 192.168.2.1/32"
#ifconfig_cxlv0_103=3D"inet vhid 2 advskew = ${MY_SKEW} pass ${PASS} = 192.168.103.1/24"
ifconfig_cxlv0_103=3D"inet = 192.168.103.1/24"
#ifconfig_cxlv0_98=3D"inet vhid 3 advskew = ${MY_SKEW} pass ${PASS} = 192.168.98.1/24"
ifconfig_cxlv0_98=3D"inet = 192.168.98.1/24"
ifconfig_cxlv0_98_ipv6=3D"inet6 = auto_linklocal"
rtadvd_interfaces=3D"cxlv0.2 = cxlv0.98"
dhclient_program=3D"/usr/local/sbin/dhcpcd"
=

#/etc/sysctl.conf
net.inet.= carp.preempt=3D1
net.inet6.ip6.dad_count=3D0
#/usr/local/etc/dhcpcd.conf
duid
pers= istent
vendorclassid
option = classless_static_routes
option interface_mtu
option = rapid_commit
require dhcp_server_identifier
slaac = hwaddr
allowinterfaces em0
waitip 4
waitip = 6
nodelay
timeout 10
interface = em0
  ia_na
  ia_pd 1/::/64 = cxlv0.2/0/64/1
  ia_pd 2/::/64 = cxlv0.98/0/64/1


= --Apple-Mail=_A9F64ED3-7010-43A7-867D-F253F3E3B109--