From nobody Fri Sep 29 15:24:17 2023
X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RxvKD2FNwz4v3FH
	for <freebsd-net@mlmmj.nyi.freebsd.org>; Fri, 29 Sep 2023 15:24:32 +0000 (UTC)
	(envelope-from kostikbel@gmail.com)
Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mx1.freebsd.org (Postfix) with ESMTPS id 4RxvKC66Q1z4H69;
	Fri, 29 Sep 2023 15:24:31 +0000 (UTC)
	(envelope-from kostikbel@gmail.com)
Authentication-Results: mx1.freebsd.org;
	none
Received: from tom.home (kib@localhost [127.0.0.1])
	by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 38TFOItJ027007
	(version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO);
	Fri, 29 Sep 2023 18:24:21 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 38TFOItJ027007
Received: (from kostik@localhost)
	by tom.home (8.17.1/8.17.1/Submit) id 38TFOIHa027006;
	Fri, 29 Sep 2023 18:24:18 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f
Date: Fri, 29 Sep 2023 18:24:17 +0300
From: Konstantin Belousov <kostikbel@gmail.com>
To: John <jwd@freebsd.org>
Cc: "Freebsd-Net@FreeBSD.org" <freebsd-net@freebsd.org>
Subject: Re: kernel crash: devctl set driver -f mlx5_core6 ppt
Message-ID: <ZRbsIcTjBIvfXlnX@kib.kiev.ua>
References: <ZRZTXuCS9GS-pu4Q@FreeBSD.org>
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-net
List-Help: <mailto:freebsd-net+help@freebsd.org>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Subscribe: <mailto:freebsd-net+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-net+unsubscribe@freebsd.org>
Sender: owner-freebsd-net@freebsd.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZRZTXuCS9GS-pu4Q@FreeBSD.org>
X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00,
	DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM,
	NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home
X-Spamd-Bar: ----
X-Rspamd-Pre-Result: action=no action;
	module=replies;
	Message is reply to one we originated
X-Spamd-Result: default: False [-4.00 / 15.00];
	REPLY(-4.00)[];
	ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]
X-Rspamd-Queue-Id: 4RxvKC66Q1z4H69

On Fri, Sep 29, 2023 at 04:32:30AM +0000, John wrote:
> Hi Folks,
> 
>    Working against 13.2-STABLE.
> 
>    I have a chance to get some bhyve VMs running on new hardware
> with Mellanox 100Gb/s cards. After creating VF entries with iovctl
> at boottime, a devctl command to detach the mlx5 driver and attach
> the ppt driver causes the kernel to crash:
> 
>  # devctl set driver -f mlx5_core6 ppt
> 
> backtrace here: https://people.freebsd.org/~jwd/mlx5.dump.txt
What is the line number for pci_iov_detach_method+0x5e?
Better, load vmcore into debugger and get the backtrace from kgdb.

> 
>    If I create the VF in ppt mode I can correctly detach
> the ppt driver and attach the mlx5 driver.
> 
>    Also of note, if multiple VFs are created and a single
> VF is targeted for the detach operation, all VFs are operated
> on. It seems the VFs are not seen as individual entities
> but a group of children in detach_method().