From nobody Wed Aug 09 13:00:23 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RLVXY3NSnz4pwqc for ; Wed, 9 Aug 2023 13:00:29 +0000 (UTC) (envelope-from sm@codenetworks.net) Received: from relayout02-q02.dominioabsoluto.net (relayout02-q02.dominioabsoluto.net [217.116.26.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RLVXX0DKPz3fw7 for ; Wed, 9 Aug 2023 13:00:28 +0000 (UTC) (envelope-from sm@codenetworks.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=codenetworks.net header.s=domabs header.b=pETPz1mP; spf=pass (mx1.freebsd.org: domain of sm@codenetworks.net designates 217.116.26.71 as permitted sender) smtp.mailfrom=sm@codenetworks.net; dmarc=none Received: from relayout02-redir.dominioabsoluto.net (relayout02-redir.dominioabsoluto.net [217.116.26.75]) by relayout02.dominioabsoluto.net (Postfix) with ESMTP id 4RLVXS3Gf6zlWyZ; Wed, 9 Aug 2023 15:00:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=codenetworks.net; s=domabs; t=1691586024; bh=nddpXKji41FXMSKl52mCHZ7RK4PmCH/FctYaS2PZ7A8=; h=Date:Subject:To:References:From:In-Reply-To:From; b=pETPz1mPB7RfE8t2RG7lKcjQEYa1AY+dw255Xx6M+XJV3pVbk5fv/GA4pJ3ruEEjf qm4L3D95Re3GLui2M8YSmUVvLeldws7HN4FJuQzWjmMjKw1GhMSKMQC+ydPCjBjR1u l7HrDq9Y1SBHDq/igDT7llM1Py6AzEX1vR37vPQQ= Received: from [192.168.3.100] (unknown [5.159.168.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sm.codenetworks.net) by relayout02-dsp.dominioabsoluto.net (Postfix) with ESMTPSA id 4RLVXS16HHzlX0c; Wed, 9 Aug 2023 15:00:24 +0200 (CEST) Message-ID: Date: Wed, 9 Aug 2023 15:00:23 +0200 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.13.1 Subject: Re: Bhyve NIC Passthru broken Content-Language: en-US To: skorpio dr , freebsd-current@freebsd.org, Michael Dexter References: From: Santiago Martinez In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-PostalOut-Country: IP: 5.159.168.225 | Country: ES X-PostalOut-Information: AntiSPAM and AntiVIRUS on relayout02 X-PostalOut-MsgID: 4RLVXS16HHzlX0c.A355C X-PostalOut-SpamCheck: no es spam, clean X-PostalOut-From: sm@codenetworks.net X-PostalOut-Watermark: 1692190824.33059@FMsGre/Xig970pOQl6pqDw X-Spam-Status: No X-Spamd-Result: default: False [-4.10 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RWL_MAILSPIKE_EXCELLENT(-0.40)[217.116.26.71:from]; R_SPF_ALLOW(-0.20)[+ip4:217.116.26.0/24]; RCVD_IN_DNSWL_LOW(-0.20)[217.116.26.71:from,217.116.26.75:received]; R_DKIM_ALLOW(-0.20)[codenetworks.net:s=domabs]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:16371, ipnet:217.116.24.0/21, country:ES]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org,callfortesting.org]; RCVD_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[codenetworks.net:+]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[codenetworks.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4RLVXX0DKPz3fw7 Hi there! I'm seeing the same, but at the end the machine end crashing. Seems that a complete power-off/on of the hardware clears it, until it random happen again when loading vmm and doing passthru. With 13.1p7 it was solid. I'm adding  Micheal as he probably know who is the right person to look into it. Santi On 8/3/23 06:48, skorpio dr wrote: > Hello, > > Starting a vm (FreeBSD 13.2) with passthru NIC results a constant > flood of error messages on the host (FreeBSD 14-CURRENT). > The NIC refuses to work in the vm (it works on the host if I turn off > the passthru). > This problem is persistent since 13.1-Release. > (Sidenote: the passthru mostly works under the EOL 13.0-p13 with the > same config, only occasionally showing the symptoms after multiple vm > reboots, and a host reboot solves the problem unlike in 13.1-release > 13.2-release and now 14.0-current) > > Ruling out NIC/board hardware fault and specific NIC driver problems, > I've tried with the following NICs: > - Intel I350-t4 > - Intel 82580 > - Broadcom BCM5720 > > ...in different desktop and server boards (with the latest BIOS): > - Asus M5A99X EVO R2.0 > - Asus M4A87TD EVO > - Asus KGPE-D16 > >   # freebsd-version -kru > 14.0-CURRENT > 14.0-CURRENT > 14.0-CURRENT > >   /boot/loader.conf: > vmm_load="YES" > nmdm_load="YES" > hw.vmm.amdvi.enable=1 > pptdevs="8/0/0 8/0/1" > >   # vm passthru|grep ppt0 > ppt0       8/0/0        Yes          82580 Gigabit Network Connection > >   the vm config file for vm-bhyve: > loader="bhyveload" > cpu=2 > memory=4G > disk0_type="virtio-blk" > disk0_name="ptproba.img" > # igb0: > passthru0="8/0/0" > uuid="2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77" > > vm-bhyve.log: > > Aug 03 02:25:16: initialising > Aug 03 02:25:16:  [loader: bhyveload] > Aug 03 02:25:16:  [cpu: 2] > Aug 03 02:25:16:  [memory: 4G] > Aug 03 02:25:16:  [hostbridge: standard] > Aug 03 02:25:16:  [com ports: com1] > Aug 03 02:25:16:  [uuid: 2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77] > Aug 03 02:25:16:  [debug mode: no] > Aug 03 02:25:16:  [primary disk: ptproba.img] > Aug 03 02:25:16:  [primary disk dev: file] > Aug 03 02:25:16: booting > Aug 03 02:25:16: bhyveload -c /dev/nmdm-ptproba.1A -S -m 4G -e > autoboot_delay=3 -d /root/vm-bhyve/ptproba/ptproba.img ptproba > Aug 03 02:25:22:  [bhyve options: -c 2 -m 4G -AHP -U > 2333ba09-dfe1-11ed-9ed6-9c5c8e54ad77 -u -S] > Aug 03 02:25:22:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s > 4:0,virtio-blk,/root/vm-bhyve/ptproba/ptproba.img -s 5:0,passthru,8/0/0] > Aug 03 02:25:22:  [bhyve console: -l com1,/dev/nmdm-ptproba.1A] > Aug 03 02:25:22: starting bhyve (run 1) > > The first few dozen messages are like the followings (couple of > hundred lines): > > ivhd0: EVT INTR 0 Status:0x1a EVT Head:0x0 Tail:0x10] >   [CMD Total 0x24] Tail:0x240, Head:0x240. > ivhd0:  [Event0: Head:0x0 Tail:0x10] >         [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] > ivhd0: EVT INTR 1 Status:0x1a EVT Head:0x10 Tail:0x20] >   [CMD Total 0x26] Tail:0x260, Head:0x260. > ivhd0:  [Event0: Head:0x10 Tail:0x20] >         [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] > ivhd0: EVT INTR 2 Status:0x1a EVT Head:0x20 Tail:0x30] >   [CMD Total 0x26] Tail:0x260, Head:0x260. > ivhd0:  [Event0: Head:0x20 Tail:0x30] >         [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] > > ...then the constant flood (about 35 lines per second in > /var/log/messages): > > ivhd0:  [Event63: Head:0x3e0 Tail:0x8d0] >         [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] > ivhd0:  [Event64: Head:0x3f0 Tail:0x8d0] >         [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] > ivhd0:  [Event65: Head:0x400 Tail:0x8d0] >         [INV_DTE devid:0xa0 addr:0xfdf9103300 type:0x3 tr:0] > > Any idea? > > Best