From nobody Sun Feb 06 14:04:45 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id F12F019A098B for ; Sun, 6 Feb 2022 14:04:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Js9y54sj0z4rPd for ; Sun, 6 Feb 2022 14:04:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8322C1385E for ; Sun, 6 Feb 2022 14:04:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 216E4jEH089057 for ; Sun, 6 Feb 2022 14:04:45 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 216E4jw4089056 for bugs@FreeBSD.org; Sun, 6 Feb 2022 14:04:45 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 261746] SR-IOV in guests Windows Server 2022 mlx5en not work. Date: Sun, 06 Feb 2022 14:04:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: michael.adm@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644156285; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=YvNGbPM80RqDU3Lfz/okWADjv10U+Bi3a49KZYtmWWg=; b=reEHMcV7yi2BUgj0+uh9EKX+1+v97XDfU9r9KRmZKcaYu//hPyx7Auy8aF4LHxbEpwbOlH pZofzF/vRPkftFUSs5UhOZBM8bcvfsdRUhuiEH6csgJFVZk8wHCkZaCHF6tpbi6qoIrY8a crMBbQFAKzDJZT47aVMLyxbxsxW0yoMV6DKUDkp8VKW1fuubqHar1EK5i9fUdgv3dlp4Td HPjaPgjc8NbJVcYs1F5NYUph1kD1ajtutwjSazhU1oDj9VF2d09EUIpJNz2uHcvPT9Jclf 6hZKRyZNN75lfyMu+d0Iq78Q2Sa++ErXUwWqpPBrnQ5lVNLavuMyH7Z9qZjTFA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1644156285; a=rsa-sha256; cv=none; b=KV92oE4yZVBXh9EdWmMRWQ1WoSTbtAPV6QiRxeqHVUUB97miJnhFUQeMZ3Razrf3wDezl2 nhvAngxR0kyN4CWcS0y8XafVH94Hx/vXdwAvVA5GoIk9xBov1qVTyrcHchHHQ1MONjwIgR ST52iFKtIRGX0Y0V7STgj66mA5i8gSkt3Z4Izhc233e+kTYBLC7t4fjGlzsRKiFyaIX+3q +69Q77PgcW34pjnNOL1ai/9cwdR2/jjHrvLuwKhbhqVjXkTzzlfMG/svY8/8A7GPBu7yJk VydLigyXpnGctIx0ARCqsLfwopt0WCDSAPSDBI+HCiht5D2wkZ3CrNYynFkcJg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261746 Bug ID: 261746 Summary: SR-IOV in guests Windows Server 2022 mlx5en not work. Product: Base System Version: CURRENT Hardware: amd64 OS: Any Status: New Severity: Affects Many People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: michael.adm@gmail.com Hypervisor: Windows Server 2022 (21H2, 20348.502) Network adapters: 1. Mellanox ConnectX-3 EN NIC for OCP; 10GbE; dual-port SFP+; PCIe3.0 x= 8; IPMI disabled; R6 (Firmware version: 2.42.5000, Driver version: 5.50.14740.= 1) 2. Mellanox ConnectX-4 Lx - HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter (Firmware version: 14.31.1200, Driver version: 2.80.25134.0)=20 Guest: FreeBSD-14.0-CURRENT-amd64-20220203-e2fe58d61b7-252875-disc1.iso Generation: 2 (Configuration Version: 10.0) No changes were made (installing the system out of the box). FreeBSD 14.0-CURRENT #0 main-n252875-e2fe58d61b7: Thu Feb 3 05:57:35 UTC 2= 022 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC a= md64 . . . VT(efifb): resolution 1024x768 Hyper-V Version: 10.0.20348 [SP0] =20 Features=3D0x2e7f PM Features=3D0x20 [C2] Features3=3D0xe0bed7b2 Timecounter "Hyper-V" frequency 10000000 Hz quality 2000 CPU: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz (2597.02-MHz K8-class CPU) Origin=3D"GenuineIntel" Id=3D0x406f1 Family=3D0x6 Model=3D0x4f Steppi= ng=3D1 =20 Features=3D0x1f83fbff =20 Features2=3D0xfefa3203 AMD Features=3D0x2c100800 AMD Features2=3D0x121 Structured Extended Features=3D0x1c2fb9 Structured Extended Features3=3D0xbc000400 XSAVE Features=3D0x1 Hypervisor: Origin =3D "Microsoft Hv" real memory =3D 8388608000 (8000 MB) avail memory =3D 8083980288 (7709 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 hardware threads random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 irqs 0-23 Launching APs: 6 5 7 3 4 2 1 Timecounter "Hyper-V-TSC" frequency 10000000 Hz quality 3000 random: entropy device external interface kbd0 at kbdmux0 efirtc0: efirtc0: registered as a time-of-day clock, resolution 1.000000s smbios0: at iomem 0xf7fd8000-0xf7fd801e smbios0: Version: 3.1, BCD Revision: 3.1 aesni0: acpi0: atrtc0: port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 acpi_syscontainer0: on acpi0 vmbus0: on acpi_syscontainer0 vmgenc0: on acpi0 vmbus_res0: irq 5 on acpi0 Timecounters tick every 10.000 msec usb_needs_explore_all: no devclass vmbus0: version 4.0 hvet0: on vmbus0 Event timer "Hyper-V" frequency 10000000 Hz quality 1000 hvheartbeat0: on vmbus0 hvkvp0: on vmbus0 hvshutdown0: on vmbus0 hvtimesync0: on vmbus0 hvtimesync0: RTT hvvss0: on vmbus0 storvsc0: on vmbus0 hvkbd0: on vmbus0 hn0: on vmbus0 hn0: Ethernet address: 00:15:5d:d0:8b:40 hn1: on vmbus0 hn0: link state changed to UP hn1: Ethernet address: 00:15:5d:d0:8b:41 hn1: link state changed to UP root@frw05v04:~ # ifconfig . . . hn0: flags=3D8843 metric 0 mtu 1500 options=3D8051b ether 00:15:5d:d0:8b:40 inet 172.27.0.24 netmask 0xffffff00 broadcast 172.27.0.255 media: Ethernet autoselect (10Gbase-T ) status: active nd6 options=3D29 hn1: flags=3D8843 metric 0 mtu 1500 options=3D8051b ether 00:15:5d:d0:8b:41 inet 172.27.172.24 netmask 0xffffff00 broadcast 172.27.172.255 media: Ethernet autoselect (10Gbase-T ) status: active nd6 options=3D29 We make sure that both the first and second virtual network adapters (hn0 a= nd hn1) have a network connectivity. All ok. Enable SR-IOV on the first network adapter: hn0: got notify, nvs type 128 pcib0: on vmbus0 pci0: on pcib0 pci0: at device 2.0 (no driver attached) root@frw05v04:~ # kldload mlx4en mlx4_core0: at device 2.0 on pci0 mlx4_core: Mellanox ConnectX core driver v3.7.0 (July 2021) mlx4_core: Initializing mlx4_core mlx4_core0: Detected virtual function - running in slave mode mlx4_core0: Sending reset mlx4_core0: Sending vhcr0 mlx4_core0: HCA minimum page size:512 mlx4_core0: Timestamping is not supported in slave mode mlx4_en mlx4_core0: Activating port:1 mlxen0: Ethernet address: 00:15:5d:d0:8b:40 mlx4_en: mlx4_core0: Port 1: Using 8 TX rings mlx4_en: mlx4_core0: Port 1: Using 8 RX rings mlx4_en: mlxen0: Using 8 TX rings mlx4_en: mlxen0: Using 8 RX rings mlxen0: link state changed to DOWN hn0: link state changed to DOWN mlx4_en: mlxen0: Initializing port mlxen0: tso6 disabled due to -txcsum6. hn0: disable IPV6 mbuf hash delivery mlxen0: link state changed to UP mlx4_core0: going promisc on 1 hn0: link state changed to UP We make sure that there is a network connection through the first network adapter. Yes everything is OK. Enable SR-IOV on the second network adapter: hn1: got notify, nvs type 128 pcib1: on vmbus0 pci1: on pcib1 mlx5_core0: at device 2.0 on pci1 mlx5: Mellanox Core driver 3.7.0 (July 2021) mlx5_core0: WARN: mlx5_init_once:962:(pid 0): Unable to find vendor specific capabilities mce0: Ethernet address: 00:15:5d:d0:8b:41 mce0: link state changed to DOWN hn1: link state changed to DOWN mlx5_core0: WARN: mlx5_fwdump_prep:94:(pid 0): Unable to find vendor-specif= ic capability, error 2 mce0: ERR: mlx5e_ioctl:3542:(pid 0): tso6 disabled due to -txcsum6. mce0: link state changed to UP hn1: disable IPV6 mbuf hash delivery hn1: link state changed to UP We make sure that there is a network connection through the first network adapter. Communication has stopped through this network adapter, both from = the VM and from outside to the VM. root@frw05v04:~ # ifconfig . . . hn0: flags=3D8843 metric 0 mtu 1500 options=3D8051b ether 00:15:5d:d0:8b:40 inet 172.27.0.24 netmask 0xffffff00 broadcast 172.27.0.255 media: Ethernet autoselect (10Gbase-CX4 ) status: active nd6 options=3D29 hn1: flags=3D8843 metric 0 mtu 1500 options=3D8051b ether 00:15:5d:d0:8b:41 inet 172.27.172.24 netmask 0xffffff00 broadcast 172.27.172.255 media: Ethernet 10GBase-CR1 status: active nd6 options=3D29 mlxen0: flags=3D8a43 metri= c 0 mtu 1500 =20=20=20=20=20=20=20 options=3D8c05bb ether 00:15:5d:d0:8b:40 media: Ethernet autoselect (10Gbase-CX4 ) status: active nd6 options=3D29 mce0: flags=3D8a43 metric = 0 mtu 1500 =20=20=20=20=20=20=20 options=3D8805bb ether 00:15:5d:d0:8b:41 media: Ethernet 10GBase-CR1 status: active nd6 options=3D29 Windows message log: "FRW_05_v4": The network adapter (1EE87C8C-AD86-4162-AAB5-866E08A8045A--8C222756-00C4-4BA1-A16E-B593B25E8453) has allocated a virtual function. (Virtual Function ID: 0; LUID: 0:15455; Virtual Machine ID: 1EE87C8C-AD86-4162-AAB5-866E08A8045A) "FRW_05_v4": Virtual PCI device successfully added to virtual machine: = "Bus {56158769-787C-4343-AC67-D9C7AAE1AACA} Slot 2". PnpID =3D "PCI\VEN_15B3&DEV_1016&SUBSYS_00D31590&REV_80", FunctionType =3D PhysicalOrVirtual. (Virtual Machine ID 1EE87C8C-AD86-4162-AAB5-866E08A8045A) "FRW_05_v4": The network adapter (1EE87C8C-AD86-4162-AAB5-866E08A8045A--8C222756-00C4-4BA1-A16E-B593B25E8453) has assigned a virtual function. (Virtual Function ID: 0; LUID: 0:15455) (Virtual Machine ID: 1EE87C8C-AD86-4162-AAB5-866E08A8045A) "FRW_05_v4": The virtual machine successfully negotiated PCI virtual bus protocol version 0x10001 on "Channel {56158769-787C-4343-AC67-D9C7AAE1AACA}= ". (Virtual Machine ID 1EE87C8C-AD86-4162-AAB5-866E08A8045A) "FRW_05_v4": Guest OS enabled virtual PCI device: "Bus {56158769-787C-4343-AC67-D9C7AAE1AACA} Slot 2". (Virtual Machine ID 1EE87C8C-AD86-4162-AAB5-866E08A8045A) HPE Ethernet 10/25Gb 2-port 640FLR-SFP28 Adapter #4: VF #0 attached to = VM FRW_05_v4 was loaded with driver version: FreeBSD,mlx5_core,14.0.51,3.7.0. We turn off SR-IOV on the second network adapter - communication through it= is resumed. OK. Recompile the GENERIC kernel with an additional option: "options RATELI= MIT" and set the boot_verbose=3D"YES" option in /boot/loader.conf . . . Hyper-V Version: 10.0.20348 [SP0] =20 Features=3D0x2e7f PM Features=3D0x20 [C2] Features3=3D0xe0bed7b2 Recommends: 000e2e24 00000fff Limits: Vcpu:1024 Lcpu:1024 Int:10416 HW Features: 0000004f, AMD: 00000000 . . . TSC timecounter disables C2 and C3. TSC timecounter discards lower 1 bit(s) No TSX change made TCP_ratelimit: Is now initialized hyperv: Hypercall created . . . acpi_syscontainer0: on acpi0 vmbus0: on acpi_syscontainer0 vmgenc0: on acpi0 vmbus_res0: irq 5 on acpi0 ACPI: Enabled 1 GPEs in block 00 to 0F Device configuration finished. . . . vmbus0: intrhook vmbus_res0: walking _CRS, pass=3D0 acpi0: walking _CRS, pass=3D0 acpi0: _CRS: not found, pass=3D0 acpi_syscontainer0: walking _CRS, pass=3D0 vmbus0: decoding 3 range 0xfe0000000-0xfffffffff vmbus_res0: walking _CRS, pass=3D1 acpi0: walking _CRS, pass=3D1 acpi0: _CRS: not found, pass=3D1 acpi_syscontainer0: walking _CRS, pass=3D1 vmbus0: decoding 3 range 0xf8000000-0xfed3ffff vmbus0: fb: fb_addr: 0xf8000000, size: 0x800000, actual size needed: 0xc0000 vmbus0: allocated type 3 (0xf8000000-0xf87fffff) for rid 0 of vmbus0 vmbus0: successfully reserved memory for framebuffer starting at 0xf8000000, size 0x800000 vmbus0: vmbus IDT vector 252 vmbus0: smp_started =3D 1 vmbus0: version 4.0 hvet0: on vmbus0 Event timer "Hyper-V" frequency 10000000 Hz quality 1000 vmbus0: chan2 subidx0 offer vmbus0: chan2 assigned to cpu0 [vcpu0] vmbus0: chan7 subidx0 offer vmbus0: chan7 assigned to cpu0 [vcpu0] vmbus0: chan8 subidx0 offer hvheartbeat0: vmbus0: chan8 assigned to cpu0 [vcpu0] vmbus0: on vmbus0 hvheartbeat0: chan9 subidx0 offer chan7 update cpu0 flag_cnt to 1 vmbus0: chan9 assigned to cpu0 [vcpu0] vmbus0: chan10 subidx0 offer vmbus0: chan10 assigned to cpu0 [vcpu0] vmbus0: chan11 subidx0 offer vmbus0: chan11 assigned to cpu0 [vcpu0] vmbus0: chan14 subidx0 offer vmbus0: chan14 assigned to cpu0 [vcpu0] vmbus0: chan1 subidx0 offer vmbus0: chan1 assigned to cpu0 [vcpu0] vmbus0: chan3 subidx0 offer vmbus0: chan3 assigned to cpu0 [vcpu0] vmbus0: chan4 subidx0 offer vmbus0: chan4 assigned to cpu0 [vcpu0] vmbus0: chan5 subidx0 offer vmbus0: chan5 assigned to cpu0 [vcpu0] vmbus0: chan6 subidx0 offer vmbus0: chan6 assigned to cpu0 [vcpu0] vmbus0: chan12 subidx0 offer vmbus0: chan12 assigned to cpu0 [vcpu0] vmbus0: chan13 subidx0 offer vmbus0: chan13 assigned to cpu0 [vcpu0] vmbus0: chan15 subidx0 offer vmbus0: chan15 assigned to cpu0 [vcpu0] hvheartbeat0: gpadl_conn(chan7) succeeded hvheartbeat0: chan7 opened hvkvp0: on vmbus0 hvkvp0: gpadl_conn(chan8) succeeded hvkvp0: chan8 opened hvshutdown0: on vmbus0 hvshutdown0: gpadl_conn(chan9) succeeded hvshutdown0: sel framework version: 3.0 hvshutdown0: hvshutdown0: supp framework version: 1.0 hvshutdown0: supp framework version: 3.0 chan9 opened hvtimesync0: hvshutdown0: sel message version: 3.0 hvshutdown0: supp message version: 1.0 hvshutdown0: supp message version: 3.0 hvshutdown0: supp message version: 3.1 hvshutdown0: supp message version: 3.2 on vmbus0 hvtimesync0: gpadl_conn(chan10) succeeded hvtimesync0: sel framework version: 3.0 hvtimesync0: hvtimesync0: supp framework version: 1.0 hvtimesync0: supp framework version: 3.0 chan10 opened hvvss0: hvtimesync0: sel message version: 4.0 on vmbus0 hvtimesync0: supp message version: 1.0 hvtimesync0: supp message version: 3.0 hvtimesync0: supp message version: 4.0 hvtimesync0: RTT hvvss0: gpadl_conn(chan11) succeeded hvtimesync0: apply sync request, hv: 1644152790950713500, vm: 1220771799 hvvss0: chan11 opened storvsc0: Enlightened SCSI device detected last message repeated 1 times storvsc0: on vmbus0 storvsc ringbuffer size: 262144, max_io: 512 storvsc0: chan14 assigned to cpu0 [vcpu0] storvsc0: gpadl_conn(chan14) succeeded storvsc0: chan14 opened storvsc0: max chans 2, multi-chan capable storvsc0: chan16 subidx1 offer storvsc0: chan16 assigned to cpu0 [vcpu0] storvsc0: chan16 assigned to cpu1 [vcpu1] storvsc0: chan16 update cpu1 flag_cnt to 1 storvsc0: gpadl_conn(chan16) succeeded storvsc0: chan16 opened Storvsc create multi-channel success! (probe0:storvsc0:0:0:0): Down reving Protocol Version from 4 to 2? hvkbd0: (probe1:storvsc0:0:1:0): Down reving Protocol Version = from 4 to 2? . . . hvheartbeat0: sel framework version: 3.0 hvheartbeat0: supp framework version: 1.0 hvheartbeat0: supp framework version: 3.0 hvheartbeat0: sel message version: 3.0 hvheartbeat0: supp message version: 3.0 hvheartbeat0: supp message version: 3.1 . . . hn1: on vmbus0 hn0: link state changed to UP hn1: LRO: entry count 128 hn1: link RX ring 0 to chan15 hn1: link TX ring 0 to chan15 hn1: chan15 assigned to cpu0 [vcpu0] hn1: gpadl_conn(chan15) succeeded hn1: chan15 opened hn1: NVS version 0x60001, NDIS version 6.30 hn1: nvs ndis conf done hn1: gpadl_conn(chan15) succeeded last message repeated 1 times hn1: chimney sending buffer 6144/2560 hn1: RNDIS ver 1.0, aggpkt size 4026531839, aggpkt cnt 8, aggpkt align 8 hn1: hwcaps rev 3 hn1: hwcaps csum: ip4 tx 0x155/0x2 rx 0x155/0x2, ip6 tx 0x55/0x2 rx 0x55/0x2 hn1: hwcaps lsov2: ip4 maxsz 62780 minsg 2 encap 0x2, ip6 maxsz 62780 minsg= 2 encap 0x2 opts 0x5 hn1: hwcaps rsc: ip4 1 ip6 1 hn1: NDIS TSO szmax 62780 sgmin 2 hn1: offload csum: ip4 4, tcp4 4, udp4 4, tcp6 4, udp6 4 hn1: offload lsov2: ip4 2, ip6 2 hn1: offload rsc: ip4 2, ip6 2 hn1: offload config done hn1: 64 RX rings hn1: RSS indirect table size 128 hn1: RSS caps 0x301 hn1: RX rings offered 64, requested 8 hn1: chan24 subidx1 offer hn1: chan24 assigned to cpu0 [vcpu0] hn1: chan25 subidx2 offer hn1: chan25 assigned to cpu0 [vcpu0] hn1: chan26 subidx3 offer hn1: chan26 assigned to cpu0 [vcpu0] hn1: chan27 subidx4 offer hn1: chan27 assigned to cpu0 [vcpu0] hn1: chan28 subidx5 offer hn1: chan28 assigned to cpu0 [vcpu0] hn1: chan29 subidx6 offer hn1: chan29 assigned to cpu0 [vcpu0] hn1: chan30 subidx7 offer hn1: chan30 assigned to cpu0 [vcpu0] hn1: 8 TX ring, 8 RX ring hn1: link RX ring 1 to chan24 hn1: link TX ring 1 to chan24 hn1: chan24 assigned to cpu1 [vcpu1] hn1: gpadl_conn(chan24) succeeded hn1: chan24 opened hn1: link RX ring 2 to chan25 hn1: link TX ring 2 to chan25 hn1: chan25 assigned to cpu2 [vcpu2] hn1: gpadl_conn(chan25) succeeded hn1: chan25 opened hn1: link RX ring 3 to chan26 hn1: link TX ring 3 to chan26 hn1: chan26 assigned to cpu3 [vcpu3] hn1: gpadl_conn(chan26) succeeded hn1: chan26 opened hn1: link RX ring 4 to chan27 hn1: link TX ring 4 to chan27 hn1: chan27 assigned to cpu4 [vcpu4] hn1: gpadl_conn(chan27) succeeded hn1: chan27 opened hn1: link RX ring 5 to chan28 hn1: link TX ring 5 to chan28 hn1: chan28 assigned to cpu5 [vcpu5] hn1: gpadl_conn(chan28) succeeded hn1: chan28 opened hn1: link RX ring 6 to chan29 hn1: link TX ring 6 to chan29 hn1: chan29 assigned to cpu6 [vcpu6] hn1: gpadl_conn(chan29) succeeded hn1: chan29 opened hn1: link RX ring 7 to chan30 hn1: link TX ring 7 to chan30 hn1: chan30 assigned to cpu7 [vcpu7] hn1: gpadl_conn(chan30) succeeded hn1: chan30 opened hn1: 7 sub-channels attached hn1: setup default RSS key hn1: setup default RSS indirect table hn1: RSS indirect table size 128, hash 0x00005701 hn1: RSS config done hn1: TX agg size 6144, pkts 8, align 8 hn1: set RX filter 0x00000000 done hn1: RNDIS mtu 1500 hn1: support HASHVAL pktinfo hn1: TSO size max 62762 hn1: bpf attached hn1: Ethernet address: 00:15:5d:d0:8b:41 hn1: TSO segcnt 31 segsz 4096 vmbus0: device scan, probe and attach done hn1: link state changed to UP . . . GEOM: new disk da0 efirtc0: providing initial system time start_init: trying /sbin/init lo0: link state changed to UP hn0: set RX filter 0x00000009 done hn0: set RX filter 0x0000000d done hn1: set RX filter 0x00000009 done hn1: set RX filter 0x0000000d done hvkvp0: sel framework version: 3.0 hvkvp0: supp framework version: 1.0 hvkvp0: supp framework version: 3.0 hvkvp0: sel message version: 4.0 hvkvp0: supp message version: 3.0 hvkvp0: supp message version: 4.0 hvkvp0: supp message version: 5.0 hvvss0: sel framework version: 3.0 hvvss0: supp framework version: 1.0 hvvss0: supp framework version: 3.0 hvvss0: sel message version: 5.0 hvvss0: supp message version: 5.0 hvvss0: supp message version: 6.0 hvvss0: supp message version: 7.0 hvtimesync0: apply sample request, hv: 1644152795952966900, vm: 1644152797140655199 We make sure that there is a network connection through the first network adapter. Yes everything is OK. Enable SR-IOV on the second network adapter: hn1: got notify, nvs type 128 vmbus0: chan31 subidx0 offer vmbus0: chan31 assigned to cpu0 [vcpu0] pcib0: on vmbus0 vmbus0: allocated type 3 (0xfe0000000-0xfe0001fff) for rid 0 of pcib0 pcib0: gpadl_conn(chan31) succeeded pcib0: chan31 opened vmbus_pcib: initialize bar 0 by writing all 1s vmbus_pcib: initialize bar 1 by writing all 1s pci0: on pcib0 pci0: domain=3D3, physical bus=3D0 found-> vendor=3D0x15b3, dev=3D0x1016, revid=3D0x80 domain=3D3, bus=3D0, slot=3D2, func=3D0 class=3D02-00-00, hdrtype=3D0x00, mfdev=3D0 cmdreg=3D0x0000, statreg=3D0x0010, cachelnsz=3D0 (dwords) lattimer=3D0x00 (0 ns), mingnt=3D0x00 (0 ns), maxlat=3D0x00 (0 ns) MSI-X supports 8 messages in map 0x10 map[10]: type Prefetchable Memory, range 64, base 0xfffffffffff00000, size 20, memory disabled mlx5_core0: at device 2.0 on pci0 mlx5: Mellanox Core driver 3.7.0 (July 2021)vmbus0: allocated type 3 (0xfe0100000-0xfe01fffff) for rid 10 of mlx5_core0 mlx5_core0: Lazy allocation of 0x100000 bytes rid 0x10 type 3 at 0xfe0100000 mlx5_core0: WARN: mlx5_init_once:962:(pid 0): Unable to find vendor specific capabilities mlx5_core0: attempting to allocate 8 MSI-X vectors (8 supported) msi: routing MSI-X IRQ 24 to local APIC 0 vector 50 msi: routing MSI-X IRQ 25 to local APIC 2 vector 48 msi: routing MSI-X IRQ 26 to local APIC 0 vector 51 msi: routing MSI-X IRQ 27 to local APIC 2 vector 49 msi: routing MSI-X IRQ 28 to local APIC 0 vector 52 msi: routing MSI-X IRQ 29 to local APIC 2 vector 50 msi: routing MSI-X IRQ 30 to local APIC 0 vector 53 msi: routing MSI-X IRQ 31 to local APIC 2 vector 51 mlx5_core0: using IRQs 24-31 for MSI-X mce0: bpf attached mce0: Ethernet address: 00:15:5d:d0:8b:41 mce0: link state changed to DOWN hn1: link state changed to DOWN mlx5_core0: WARN: mlx5_fwdump_prep:94:(pid 0): Unable to find vendor-specif= ic capability, error 2 mce0: ERR: mlx5e_ioctl:3542:(pid 0): tso6 disabled due to -txcsum6. hn1: delayed initialize mce0 hn1: try bringing up mce0 mce0: link state changed to UP hn1: disable IPV6 mbuf hash delivery hn1: disable RSS hn1: RSS indirect table size 128, hash 0x00005701 hn1: link state changed to UP hn1: RSS config done hn1: reconfig RSS hn1: RSS indirect table size 128, hash 0x00005701 hn1: RSS config done Communication has stopped through this network adapter, both from the VM and from outside to the VM. Turn off SR-IOV on the second network adapter: mce0: link state changed to DOWN hn1: link state changed to DOWN hn1: link state changed to UP mlx5_core0: detached pcib0: chan31 revoked pcib0: chan31 detached pci0: detached hn1: got notify, nvs type 128 pcib0: chan31 closed pcib0: detached vmbus0: chan31 freed communication through second network adapter resumed. Any ideas? --=20 You are receiving this mail because: You are the assignee for the bug.=