From nobody Thu Dec 28 16:01:32 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T1CtQ4qJcz564jZ for ; Thu, 28 Dec 2023 16:01:34 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T1CtQ4MWhz3S4r; Thu, 28 Dec 2023 16:01:34 +0000 (UTC) (envelope-from dim@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703779294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rM0hdkIH81FFG4E9ek0mL8bHs3c46TNhFTDg45mXz9o=; b=NiEnAIT9C6o8kC6lGyeGsnFhk+RLPAaDQ298Pw5Np3gpHbOoxYkZQAGURuaSsZGrxxOdvE 8uCYq43Qddu+q8eZb4ThGI4RJux7aiSdHCyAWu67C15AuqVb3ePZfPPBzI9zg8fvyJV2PU O/vLdAzCSRtC5+RqbjqgFq2nipgV/CmJtjvXTsSxBi75/jaz10u6vx1BZbsnTcRrzwYI8o oa55IhARLWKA16RenD3L3QEUuW+c9mmNdL5J9zhKrTbKQ1QCIsoHb5wx9AilU3rNNyg0CU xehXatm6Fw+GxrOtZ2FiE/qGwqT7ELIcwkdG6hJfMsdrWrsWFmx4pMY6ZoN7hw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1703779294; a=rsa-sha256; cv=none; b=X18zD5wECnknQHFNLn0khMBVz2cfssZCTJWHD4jSKtH59CQ2HG9202DwKA+SDSmE91U3vN +h+J3C4tJU1YINaZTgUTGux00u33JinbNxO4m44ZLRPT/HmxQfZ6MkLtzXixOJsv/u5XeI 6C4E50eXMACfvZ2mtaggalgXTKlWynOkm4OSc4P7rM1WKb8T32GirJVUV1egpyinGOoUrR se0cUFWDedSFy6M+TY8VxNNZArm8ypdCeKdLwDboqAsSZgSTfN42W4vHyOrnJAQHzoEHO4 5mEI4IAivngnZbMzd3P5JbCizUw5eMHXPbciP9+YjlNj1ErCsNuL5BRBEmjF4A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1703779294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rM0hdkIH81FFG4E9ek0mL8bHs3c46TNhFTDg45mXz9o=; b=oMZlW8FctpM4fTUktjiD0+jETMiypR5FT1NUczHUGNIQZDdykV4iVTKdk2xMbtzijyPGnq pU6IFCEru4SCjM2DN6Xcd/kU6QBniVseS8BfbnVwOfXMs0RK3t+yESj4u+Yg8O7adYmY2Q d7CYYvFNarLpWg3HlyfzCoxQF2jIBw8/ckyF3m6p6oulQZkH5JDPnSKMZzUxfAIdWay+gu HqQiuHJwCm3QCCDKbTUCI8TJ2foRF05Vvmxee0aH3PEtHghAS6ReKyO0fYbHu9oRtWhF39 tJae1fD9E5SkeEM4D8ZjzTYnTgQc7yLsJYN9OurDMK1Tle2lLg/2JZN4PCTYxQ== Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "tensor.andric.com", Issuer "R3" (verified OK)) (Authenticated sender: dim) by smtp.freebsd.org (Postfix) with ESMTPSA id 4T1CtQ2rWYz4Yr; Thu, 28 Dec 2023 16:01:34 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtpclient.apple (longrow.home.andric.com [192.168.0.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 1CD6E3DB0E; Thu, 28 Dec 2023 17:01:33 +0100 (CET) Content-Type: text/plain; charset=us-ascii List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS pool hangs (live-locks?) after adding L2ARC From: Dimitry Andric In-Reply-To: Date: Thu, 28 Dec 2023 17:01:32 +0100 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <3260F369-4F60-46E6-939A-B3668F347622@FreeBSD.org> References: <88ae141a-93af-4f23-87c1-f44a99975a28@FreeBSD.org> To: "lev@freebsd.org" X-Mailer: Apple Mail (2.3731.700.6) On 28 Dec 2023, at 16:03, Lev Serebryakov wrote: >=20 > On 28.12.2023 15:16, Dimitry Andric wrote: >=20 >>>> I can not find any discussion about such problem on Internet. = Also, "live" system doesn't have these "z_int_X_Y" threads at all. >>>> I want my L2ARC, I've payed for this NVMe! >>>=20 >>> Any ideas? >> Do you know 100% sure that this new NVMe device is working correctly? = Are there any errors to be found in SMART information, dmesg or syslog? >=20 > I'm running "ssd-steadystate.fio" on this SSD now (not whole SSD but = only l2arc partition, which is 85% of whole SSD to have 15% = over-provisioning), after running `trim` on it. >=20 > Let see... >=20 > If you could advice better scenario for `fio` (with some TRIM added = for good measure?) I can run it too. If these tests all work, I would first try making a plain new zpool, = with just this one device under it, and with the same settings you are = using for the other zpools. Then mount the pool somewhere, and exercise it with whatever workload = you can find, to see if that too "locks up". If not, you may have found = some sort of problem specific to l2arc, and I would guess it is best to = take that upstream? If it does lock up, there is probably some sort of incompatibility = between your NVMe device and the FreeBSD driver. You could then try = using another driver, IIRC there are two different ones now? -Dimitry