From nobody Fri Feb 25 10:02:48 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 212E519E1132 for ; Fri, 25 Feb 2022 10:02:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K4lh824xXz4pFm for ; Fri, 25 Feb 2022 10:02:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1ECBC1DD85 for ; Fri, 25 Feb 2022 10:02:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 21PA2l0c099282 for ; Fri, 25 Feb 2022 10:02:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 21PA2lKG099281 for bugs@FreeBSD.org; Fri, 25 Feb 2022 10:02:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 262189] ZFS volume not showing up in /dev/zvol when 1 CPU Date: Fri, 25 Feb 2022 10:02:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: zedupsys@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645783368; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=kVCJBRJjYEmfe4FCHibxlFtY5E827w7++BYary8t0G8=; b=huNoWRn4ZaDIkNa6vDLfXyqVZ85GMTonw6WXc1pElVXp0reevKJCdZjXh0cjE82tilB5zu i29UTjJEGx93isW+AjjETmJ2g+5tS7J0sEI701bFYiWpu8VKqw2WP60PyPd4Utkug9JFem DQ6XGOsdWjepwihPqPsrTpx6+l3Dbou42SVrlcO8OKrFO/KaooQzeRh0cHoWl8f9tua8Np qjw0NLK/RNX13XbGIXkXN4u0oIEOPsTbjHOPCJJxBMV4gKrHKde+KrIoTG5oBfIN6cCE1u prXB+70RQY97oBQGG8dUV0yqzexQht4Xwtv0RjkABAN6Bil94AKQEFosiaz1lg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1645783368; a=rsa-sha256; cv=none; b=BvzqB9CaK9JeOP81ZeCSkd87QikjUg/zAWUZ/ovpc0ErsOMRD2dwDHbxJMJuAAjnJdElQ8 NaKejdBEdbHhIIf4XBtqPh9fksdc1BA6BWYATvC6k/1/i1iUz6bkvqF66iWs1p8+v/bQa6 5RHDf22bQGp9Fwc1loVQ4rLqvc5hHOJ8YH3YeX80/qc8sU6um0uMZv+Ko99YIeQGyPN0In D2D0j6jT26F1eImrFL1XIKMGlQ4TpD0HOdk8k1S8CvhmhzlekFAFrgprVCjCL1wiMQYc/t nKQMAN94bJq2i/dHKasgHeSZx/BwClrFQmvjP16/DRBaqZM4gAzDFE13RzydtA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D262189 Bug ID: 262189 Summary: ZFS volume not showing up in /dev/zvol when 1 CPU Product: Base System Version: 13.0-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: zedupsys@gmail.com I have found 100% repeatable problem on 4+ different setups, with ZFS zvol = not showing up in /dev/zvol until system reboot. In a way this is a continuatio= n of investigation for problem reported at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D261059, where i had my suspicion that there are some ZFS concurrency issues. Requirements to reproduce (as i have tested), latest BSD as of now (13.0-RELEASE-p7 or 13.0-RELEASE) machine must have 1CPU (or 1 vCPU !IMPORTANT!). RAM seems not to matter, i have tested with 16G, 2G, 1G setup= s. Example will be for basic ZFS install, default options from DVD installer, automatic partitioning (just next-next install). Ensure that before running there is no ZVOLs and /dev/zvol directory does not exist (not mandatory, bug exists even if there are zvols, just easier to detect if there aren't any). To trigger the bug, run shell script below (adjust script preamble of zpool= /zfs dataset creation/destruction, name as necessary): /bin/sh name_pool=3Dzroot/stress # zpool create -f $name_pool /dev/ada1 # or # zfs create $name_pool zfs set mountpoint=3Dnone $name_pool # zfs destroy -r $name_pool seq 1 100 | while read i; do zfs create -o volmode=3Ddev -V 1G $name_pool/data$i dd if=3D/dev/zero of=3D/dev/zvol/$name_pool/data$i bs=3D1M done You will see error output like or similar at some point in loop: dd: /dev/zvol/zroot/stress/data1: No such file or directory Now to validate run: ls /dev/zvol ls: /dev/zvol: No such file or directory zfs list # .. output containing all created ZVOLS .. After reboot, zvols show up in /dev/zvol as expected (as should have been t= he case after create). More details and observations on different environments: 1. 100% reproducible inside VirtualBox 6.0 VM (default FreeBSD settings, 13.0-RELEASE, default ZFS install), 1vCPU, 1GB RAM. Zvols are created on the same Zpool where BSD is installed. 2. 100% reproducible inside XEN 4.15 DomU, 1vCPU, 2GB RAM. FreeBSD installed on ada0 UFS, zpool created on /dev/ada1 whole disk, zvol directly (name_pool=3Dzroot) without hierarchy. 3. 100% reproducible inside XEN 4.15 Dom0, 1vCPU, 16GB RAM, 13.0-RELEASE-p7. FreeBSD installed on separate /dev/gpt/label1 (ada0) disk, Zpool on ada1 gpt partitioned. 4. 100% reproducible on physical hardware, in BIOS disable all CPU cores, exce= pt one. 16GB RAM, Xeon CPU. Observations: This bug seems to be CPU count related. If 2 CPUs are available, there will= be around 30% not created /dev/zvol devices, If 4 CPUs, then around 15% or less (percentage calculations are not exactly calculated, but shows CPU role, concurrency). I do not have more CPUs in my testing hardware, but it seems = that the more there are, the less probable that this bug will manifest itself. For script part "seq 1 100", on single CPU is far too much, it is enough to= be "1 to 15" to see enough, for more CPU, higher count is better, since someti= mes all ZVOLs are created. After restart ZVOLs are always showing up in /dev/zvol. This works for rebo= ot and manual import as well. If zpool is on separate disk not where FreeBSD is installed, zpool export, zpool import results that ZVOLs showing up in /dev/zvol. There is no difference if ZVOL is sparse volume, created with -s flag. On 4 CPU setup, sometimes i have noticed errors like this in serial console: g_dev_taste: g_dev_taste(zvol/test/data22) failed to g_attach, error=3D6 For me this seems suspicious, since volmode=3Ddev and g_dev_taste should not trigger g_attach on such devices (volmode=3Ddev), am i right? What this err= or code mean, is it from errno.h, "#define ENXIO 6 /* Device not configured */= "? Maybe those are related to this bug, maybe not. If i dd /dev/zero on block device and detach/attach it, this does not trigg= er g_dev_taste error. I have seen 6 similar reported bugs with ZVOL not showing up. Though those = were related to different (clone, send and recv) commands and seemed outdated. I will investigate those as well, and link them in my next comments. Maybe th= ey have similar cause, but did not look like duplicates. At the moment this is as far as i am able to dig onto this. --=20 You are receiving this mail because: You are the assignee for the bug.=