From nobody Sat Aug 26 22:31:33 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RYBPd68Q0z4r7Dl for ; Sat, 26 Aug 2023 22:31:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RYBPd3j7lz3DCX for ; Sat, 26 Aug 2023 22:31:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693089093; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=XgSj7YFDSehw7ISKXT7I3I/IZeRNZ7WlolT6pu2qBgA=; b=qzAKpcOqKzuY1ecr4/Ke9SFBsy3BRNEfqQQqGI8U1s+JQiqETJs1BPC/AtotL4BqLCQKsB JtGpmsJDqE2A+jipkPoLaWla7Q4R3zOpMZHMRbWZ3yoEzhP353E9/wblBqOvLJyqUQcNGy UO/CCS7rJPTHdzuWq15Px3e/0dVXNFinHDS35ilQXS/qAUPrgMgfpThJ/s77V+gGeGYghG zrWl5U4o8R4JbhxshYGR+I3s7xNIR9FUNvReu7GX5yuhxRatp6wIbHW24GhpriKiMkakn3 sOrZyCJZOnNkPhFJ0osAnBk8J8dXahER1KyS3E4c2UArudaSN74ABQoCiT20PQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693089093; a=rsa-sha256; cv=none; b=xUNRHT6hmu5BZ/8s+8NfEq0fFRwfGjpiIzuhRoKqLudLFP6A8Nao+vpTV6EnI7CnLM5ygT Mbp+m+D9EwzJ3nw/GTgr2YKqI2wEr1Wur5MgsTpShFmiTfurU6WqCWSfwmbt0CFD1xughh k14cewUIH5l9GsIz2TZ9z1TziWaaFM+5Q+Y98GT2w8mv2M0fFxf9TS48eeuUgy0jaAe+IG 3H6kZMirJgrh5OUbB744MQh2l4Gqpfs+hk289LOjqd6jaJMW9+jdcyueSgn87VT9mjxs5q 2uf+OfaC4Noy0EmGxA2eiwZhq+7c7MFWifOUVVnqCtAX+EaD3LawH3llSA2n5A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RYBPd2pHTzsWQ for ; Sat, 26 Aug 2023 22:31:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 37QMVXDV069793 for ; Sat, 26 Aug 2023 22:31:33 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 37QMVXeU069792 for virtualization@FreeBSD.org; Sat, 26 Aug 2023 22:31:33 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 273372] SR-IOV Networking in Bhyve Causes Chelsio T520-SO-CR to Fail on Host, Kernel Panic if Reset Date: Sat, 26 Aug 2023 22:31:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: mark@markmcb.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D273372 Bug ID: 273372 Summary: SR-IOV Networking in Bhyve Causes Chelsio T520-SO-CR to Fail on Host, Kernel Panic if Reset Product: Base System Version: 13.2-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Many People Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: mark@markmcb.com If I setup SR-IOV for bhyve using a Chelsio T520-SO-CR network adapter, sta= rt the install process for a FreeBSD guest, but shortly (maybe a minute) after= I load the driver for the pci device in the guest, the network card stops responding in the guest and host. # Relevant /boot/loader.conf vmm_load=3D"YES" nmdm_load=3D"YES" t5fw_cfg_load=3D"YES" if_cxgbe_load=3D"YES" if_cxgbev_load=3D"YES" # Relevant /etc/rc.conf iovctl_files=3D"/etc/iov/cxl1.conf" vm_enable=3D"YES" vm_dir=3D"zfs:zapps/bhyve" vm_list=3D"" vm_delay=3D"5" # Relevant /etc/iov/cxl1.conf PF { device : "cxl1";=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 num_vfs : 14; } DEFAULT { passthrough : false; } # ... # VFs for bhyve VF-10 { mac-addr : "aa:88:44:00:02:20"; passthrough : true; } # ... ### pciconf -lvbc in host ppt0@pci0:2:0:49: class=3D0x020000 rev=3D0x00 hdr=3D0x00 vendor=3D0x1= 425 device=3D0x5807 subvendor=3D0x1425 subdevice=3D0x0000 vendor =3D 'Chelsio Communications Inc' device =3D 'T520-SO Unified Wire Ethernet Controller [VF]' class =3D network subclass =3D ethernet bar [10] =3D type Memory, range 32, base 0xdd00a000, size 4096, enabl= ed bar [18] =3D type Memory, range 32, base 0xdd060000, size 32768, enab= led bar [20] =3D type Memory, range 32, base 0xdd194000, size 8192, enabl= ed cap 10[70] =3D PCI-Express 2 endpoint max data 256(2048) FLR NS max read 512 link x0(x8) speed 0.0(8.0) cap 11[b0] =3D MSI-X supports 8 messages Table in map 0x20[0x0], PBA in map 0x20[0x8000] cap 05[50] =3D MSI supports 32 messages, 64 bit, vector masks ecap 0001[100] =3D AER 2 0 fatal 0 non-fatal 0 corrected ecap 000e[140] =3D ARI 1 ecap 0017[150] =3D TPH Requester 1 Use vm install freebsd-test FreeBSD-13.2-RELEASE-amd64-bootonly.iso Config line for passthru: passthru0=3D"2/0/49" Connect with vm console freebsd-test Installer starts, choose shell. ### pciconf -lvbc in guest, prior to driver load none0@pci0:0:5:0: class=3D0x020000 rev=3D0x00 hdr=3D0x00 vendor=3D0x1= 425 device=3D0x5807 subvendor=3D0x1425 subdevice=3D0x0000 vendor =3D 'Chelsio Communications Inc' device =3D 'T520-SO Unified Wire Ethernet Controller [VF]' class =3D network subclass =3D ethernet bar [10] =3D type Memory, range 32, base 0xc000e000, size 4096, enabl= ed bar [18] =3D type Memory, range 32, base 0xc0000000, size 32768, enab= led bar [20] =3D type Memory, range 32, base 0xc000c000, size 8192, enabl= ed cap 10[70] =3D PCI-Express 2 endpoint max data 256(2048) FLR NS max read 512 link x0(x8) speed 0.0(8.0) cap 11[b0] =3D MSI-X supports 8 messages Table in map 0x20[0x0], PBA in map 0x20[0x8000] cap 05[50] =3D MSI supports 32 messages, 64 bit, vector masks # kldload cxlv t5vf0: mem 0xc000e000-0xc000efff,0xc0000000-0xc0007fff,0xc000c000-0xc000dfff at device= 5.0 on pci0 t5vf0: 1 ports, 2 MSI-X interrupts, 3 eq, 2 iq cxlv0: on t5vf0 cxlv0: 1 txq, 1 rxq (NIC) ### pciconf -lvbc in guest, after driver load t5vf0@pci0:0:5:0: class=3D0x020000 rev=3D0x00 hdr=3D0x00 vendor=3D0x1= 425 device=3D0x5807 subvendor=3D0x1425 subdevice=3D0x0000 vendor =3D 'Chelsio Communications Inc' device =3D 'T520-SO Unified Wire Ethernet Controller [VF]' class =3D network subclass =3D ethernet bar [10] =3D type Memory, range 32, base 0xc000e000, size 4096, enabl= ed bar [18] =3D type Memory, range 32, base 0xc0000000, size 32768, enab= led bar [20] =3D type Memory, range 32, base 0xc000c000, size 8192, enabl= ed cap 10[70] =3D PCI-Express 2 endpoint max data 256(2048) FLR NS max read 512 link x0(x8) speed 0.0(8.0) cap 11[b0] =3D MSI-X supports 8 messages, enabled Table in map 0x20[0x0], PBA in map 0x20[0x8000] cap 05[50] =3D MSI supports 32 messages, 64 bit, vector masks Shortly after loading the driver, I lose networking on the host. dmesg shows nothing after event # ifconfig looks normal cxl0: flags=3D8843 metric 0 mtu 1500 =20=20=20=20=20=20=20 options=3D6ec07bb ether 00:07:43:36:bc:80 inet 10.0.1.201 netmask 0xffffff00 broadcast 10.0.1.255 media: Ethernet 10Gbase-Twinax status: active nd6 options=3D29 # ifconfig cxl0 down # ifconfig cxl0 up # Aug 26 15:21:11 core18 kernel: t5nex0: command 0x16 in mbox 4 timed out (0x4014c010). Aug 26 15:21:11 core18 kernel: t5nex0: mbox 4 cmdsent 16a0094400000001 05dc050000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 26 15:21:11 core18 kernel: t5nex0: mbox 4 current 16a0094400000001 05dc050000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Aug 26 15:21:11 core18 kernel: t5nex0: encountered fatal error, adapter sto= pped (1). --=20 You are receiving this mail because: You are the assignee for the bug.=