From nobody Sat Aug 19 23:27:26 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RSvzf6g8rz4qMMB for ; Sat, 19 Aug 2023 23:27:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-8.consmr.mail.gq1.yahoo.com (sonic308-8.consmr.mail.gq1.yahoo.com [98.137.68.32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RSvzf0QkXz3Wv8 for ; Sat, 19 Aug 2023 23:27:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=DRGarmtQ; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.32 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692487659; bh=es5JdmrdzOfdbgG9AcmbfIwAPVOkI7wOD3mKxq3mLL4=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=DRGarmtQ+LEgI4RNS8/UYxllQ22jvN/4dknnqcSDldqvAw9FlhE+3+kad6Xvq/yB2vfRfvxALDSln6Tc3ncgVO853mJVeKN8b/4jOvLoNH2veo1YidZiGlmwQUQwDlqJP6ujoumq4HxlxTf6JDDA0+uwir0Oezd0teN7XJskQVXdGJc01ut7hFe7V3KM6ySBvFRnLwcjIHqEOPzG3LflgQS1w6FOhSBlK6jncH0U/IRFLhgw+2N3ahVeRt7wphs40oDlxNjbxVZAxkKu8j5YGIFGiiXtPy1ze3Kh8elJhqk2CqB7YBqHyy6xfvmSd4BlGw2AgKM3Vz90JMijk75g2Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692487659; bh=6XoOtFcwuRh09Vw8/HYPjGPz2jY/IVIMNCDqJfGUXL/=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=TWD15C6gIyp3pj2wRaIQB6n+04yBXxjWNh7dx6tAenx1H26gGvHUpLic+6nf2dRuKeOsj2RyxIwrrEkNItHvYYyoAobvXVhVRUEhPUFmqBgonQnQyu22sd2yrLVDQ0NQH2E2z1l01N+NyOtVaA7026l5lh5lzFZDqEFm233kn/smVE92bYSvr7GIjqAH/WtVXYrJ6UFkqbwgBM+R3Ps2PPbXi4UhBCwXhKI7mpjss8MDoMSY9UOEbrbylTl6l5pSHds9YVOfxzJ+4E8kjYrtQ3aBv5LBGF+wYLpORsXcmi9E+BTWtodyqv4c5opk0V11jIk+M+chmbt3taTxr0Q+gw== X-YMail-OSG: fsd2u6cVM1katebCF8jY.vJdOJb3CPmS0zN4Ee.hgsVonEkAtZKo3qJWl0xYPld GZhBxVo4Fb.ZDT6K_wgip1x0xo6yOFEptebg3ASNpb.y8ZmRIfAt1fvwW0P2xZRi.iEZAarQkTGh rCVzqUxqhPRNQAs4k.tOh_NybrOMHWeCfCoQTsx7157R64n8M6vGPXr6cvm5OXoHgmh9wLj3EexQ Z5gQUmYwLVdHksXLazSP6rLosXYaCzRfNRFb_Va1Hub_kQzgDKOvifqD1yjlQyEM014mbp1kwVnf oTygI6ENkql0VAg.cS9_QfF6QH_Jm.6XWgfd15mgkDM9r0CqXQ0VwK4nnmzrKeQ_zR4F3wa.lH9M bou9_kUNC4tWDdo9weRj.T9qsRXAPiTqDzMC3HgSCVrOC1H3HJqkG7Yj0UnODY4dxV0VJn8gWq0z .7xWqRg5BRJenvdW7MRSvcUndDyTWxgJRflecf8ytAC5ylCsa1SF1BjUhNrT2_qO2gXY4skX3RNs IOk11MLlwkIWv5IQGmcRCJqcytLe_bg8iZbt.h6W5Puut10LnmchbdSGzA3.n4bPaUvkrmIq6vW0 YXD.mAwm700xFiVLm6.2gq2LWxNAZeiEiAzIwE.leADHrRcR3TzMSlYz67xo_z.kIHjxNdJbQtKW 5LoQqLh4DtBYXoFdaQwFanuIV0SuojYVY2Q4UpkMoGKA4rdiLgO4vn0BIuOTQ4ZlWLyy_Me0fRts ATMTlTdF_Qzb.wlec4MX4D2M2eGyW.VKIN1IXev4gwqnUAgyJ1sXuIthtpLWqLg8Pim4OJdbPzUl zGeKZeWdjil.y742bMudCs6zFAHAwuwKiYNfNmwZgIexZNg2HCRp45WkGEjDX0aAy20ggpQNvZfR d9wiLDlAaxRxTlSD4bwQ9VfBKpDacAGY_3c6cwiJhR0kHpvSlFShGBkYoIG9Fcx6WiRARFkVWftr kZDImxiy8qYqK8xcqTUDa1Tld5DHcFWggYM1Ju9RYeEpciF4iVGEQjeEGyVXKy2z_Zi7iDvGQIgK vEILns91HyPylSIhbcrEn6h41TJo8LsuO4_2NUzdjF91UyoPGtk6NLqH550sGDhaZGApSu1NZq0z 3U_4ULKQqm9Sbl9qIEDTZgTNBK7x59MoudIxkVsbe9c2O8d3Q240AJGZ5SRPGCqzkShAqFN.EwzG 89BuDqlJ9MvHneLaNTrKbZ2jLFu0vCgENE4SAx1heDhVl230nRxZAJ1O162drF.bqWzNPTXgyYQW IQFT27oYPKAN_vzAaiC3_2N.KoYqW1kH0OSpxF7DmAF0s.4DG_62JqPwEUmQGxqkirnOH5.gQ_EX v21Ip2zy__FYReKsgOjhWF9_YgZ02lWaEqfCcWplXso1Rmfl5oGYnvUYjNrV9pQwFb.kf2UcDLkK BS0sxHHoS2NruicVxyHmbbrBhHFP_xv1Ar4tt7bO7OxEBpRD4KBCVdG2sMDbmDe2twS8x8MaDQdp MPmWiwnFa2K8Wu4yC41BxbbbuBx_DcB35NqwyhfV3zCWjBEHGHN1Yf2OvvjaQp7ck8kTuygmjbxT cQ_405g96qn._zbtkOQehZ1xBCuh8mQG7yfljfDxjDpRlYakVTTX3ZKhGTFIyJPFWBUtZuEtXGwh VnXgCCo_PdG8s_uUAA0ETJyC6iXa3ElmDddoyb7vipaC1KNeVa_5mxTJOaJBHFhAB2IUrzTl8u7d gWUkWPPRibpAm3oZv3rHCIO1KqaIGpdhnP9RYfHcIaoanBaw6e2TM0K2y2eClcwm8YURI7pZAwqs xIJ0vas2AHadVjh7Cwk1rtGFbZUK3SxJ7DcovzV5gKlrSwAb4E.uOxuTlz7z1Clm6Pz_KyJi.h_R __cgDYN7TV287uO8ofEFAW_V9cno9Cs8.RSatLZIg9fRa1uyJN40jLcHCr11_LlqSmA.DdRC2a9z xRbQkwSzcjKowsJmqXVSdMmhI3LYJgwJs.GCmxAxBmdxQYrRJPDu9Ky7t5UBOaDspD.1X4RWHaF7 OJhtDbIU1osu5srwPtemRfRzUMUnpdjp0jdql_ObMvqzPg1I48FL93wwyjdovWqCmZPuULefDGPj 8ViBlVmMF0YvIyiE15J68f7xNINgTHjcw9UzGdYHHjCoTqfgfQBVU9UdD8gmdCeNX0KGhPLOb4Jb LaZBz69DsOXM43HPTR4gph4ZGLfGCqJO8srur71Txmm48krW5cXZ3CmdkUcbwhCSzKg6ddXiyhTQ 92tP3tJqHxPhoqUj2R.IrfZ7__fv3lNma4OuH3ulLD2HeS.wG5m4R56UN5uYd X-Sonic-MF: X-Sonic-ID: 73496822-1ac1-4f78-8e00-283fb58073e6 Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Sat, 19 Aug 2023 23:27:39 +0000 Received: by hermes--production-ne1-7b767b77cc-9p4qj (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 10bb64efe3f1897f813ca28fb48133b0; Sat, 19 Aug 2023 23:27:38 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS deadlock in 14 [USE_TMPFS=no poudriere messed up from the start, lots of "vlruwk"] Date: Sat, 19 Aug 2023 16:27:26 -0700 References: <59FCB309-4A55-4924-98C4-7ACCA70FD299@yahoo.com> <0F2C42B4-36FF-443A-A174-5B0CC57C4FC7@yahoo.com> <3AA253E3-C4F0-4AA3-9C37-D77E7527A458@yahoo.com> To: Current FreeBSD In-Reply-To: Message-Id: <8D0C1422-CE60-4266-8051-2296C3E9B7D7@yahoo.com> X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.32:from]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.32:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RSvzf0QkXz3Wv8 On Aug 19, 2023, at 15:41, Mark Millard wrote: > On Aug 19, 2023, at 13:41, Mark Millard wrote: >=20 >> [I forgot to adjust USE_TMPFS for the purpose of the test. >> So I'll later be starting over.] >>=20 >> . . . >=20 > I finally got around to starting a from-scratch bulk -a > again (based on USE_TMPFS=3Dno this time). This is with > 15107.patch and 15122.patch applied. This is a non-debug > kernel experiment. >=20 > Interstingly it got: >=20 > [00:01:34] [01] [00:00:00] Builder starting > [00:01:57] [01] [00:00:23] Builder started > [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 > [00:03:09] [01] [00:01:12] Finished ports-mgmt/pkg | pkg-1.20.4: = Success > [00:03:21] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 > [00:03:21] [02] [00:00:00] Builder starting > [00:03:21] [03] [00:00:00] Builder starting > [00:03:21] [04] [00:00:00] Builder starting > [00:03:21] [05] [00:00:00] Builder starting > [00:03:21] [06] [00:00:00] Builder starting > [00:03:21] [07] [00:00:00] Builder starting > [00:03:22] [08] [00:00:00] Builder starting > [00:03:22] [09] [00:00:00] Builder starting > [00:03:22] [10] [00:00:00] Builder starting > [00:03:22] [11] [00:00:00] Builder starting > [00:03:22] [12] [00:00:00] Builder starting > [00:03:22] [13] [00:00:00] Builder starting > [00:03:22] [14] [00:00:00] Builder starting > [00:03:22] [15] [00:00:00] Builder starting > [00:03:22] [16] [00:00:00] Builder starting > [00:03:22] [17] [00:00:00] Builder starting > [00:03:22] [18] [00:00:00] Builder starting > [00:03:22] [19] [00:00:00] Builder starting > [00:03:22] [20] [00:00:00] Builder starting > [00:03:22] [21] [00:00:00] Builder starting > [00:03:22] [22] [00:00:00] Builder starting > [00:03:22] [23] [00:00:00] Builder starting > [00:03:22] [24] [00:00:00] Builder starting > [00:03:22] [25] [00:00:00] Builder starting > [00:03:22] [26] [00:00:00] Builder starting > [00:03:22] [27] [00:00:00] Builder starting > [00:03:22] [28] [00:00:00] Builder starting > [00:03:22] [29] [00:00:00] Builder starting > [00:03:22] [30] [00:00:00] Builder starting > [00:03:22] [31] [00:00:00] Builder starting > [00:03:22] [32] [00:00:00] Builder starting > [00:03:30] [01] [00:00:09] Finished print/indexinfo | indexinfo-0.3.1: = Success > [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 >=20 > and is still that way minutes later. >=20 > ^T shows: >=20 > [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 > load: 13.02 cmd: sh 2187 [vlruwk] 570.19r 0.62u 38.60s 9% 3948k > #0 0xffffffff80b7701b at mi_switch+0xbb > #1 0xffffffff80bc976f at sleepq_timedwait+0x2f > #2 0xffffffff80b76770 at _sleep+0x1d0 > #3 0xffffffff80c5b435 at vn_alloc_hard+0x2a5 > #4 0xffffffff80c50b72 at getnewvnode_reserve+0x92 > #5 0xffffffff829b9b12 at zfs_zget+0x22 > #6 0xffffffff829a6a8d at zfs_dirent_lookup+0x16d > #7 0xffffffff829a6b5f at zfs_dirlook+0x7f > #8 0xffffffff829b6410 at zfs_lookup+0x350 > #9 0xffffffff829b182a at zfs_freebsd_cachedlookup+0x6a > #10 0xffffffff80c36a0d at vfs_cache_lookup+0xad > #11 0xffffffff80c45141 at vfs_lookup+0x581 > #12 0xffffffff80c44238 at namei+0x238 > #13 0xffffffff80c63b5e at kern_statat+0xee > #14 0xffffffff80c64237 at sys_fstatat+0x27 > #15 0xffffffff81049a79 at amd64_syscall+0x109 > #16 0xffffffff8101f11b at fast_syscall_common+0xf8 > [main-amd64-bulk_a-default] [2023-08-19_15h14m10s] [parallel_build:] = Queued: 34435 Built: 2 Failed: 0 Skipped: 35 Ignored: 358 = Fetched: 0 Tobuild: 34040 Time: 00:10:52 > ID TOTAL ORIGIN PKGNAME PHASE PHASE = TMPFS CPU% MEM% > [01] 00:07:29 devel/gettext-runtime | gettext-runtime-0.22 build = 00:06:32 25.4% 0% > [00:11:25] Logs: = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-08-19_1= 5h14m10s >=20 > Note the 3:31->11:25 . >=20 > Top is showing lots of "vlruwk". For example: >=20 > 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 = 0.00% cpdup -i0 -o ref 32 > 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 = 0.01% cpdup -i0 -o ref 31 > 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 = 0.01% cpdup -i0 -o ref 30 > 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 = 0.01% cpdup -i0 -o ref 29 > 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 = 0.01% cpdup -i0 -o ref 28 > 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 = 0.00% cpdup -i0 -o ref 27 >=20 > In other words, it is messed up from the start, not > just later. >=20 > It does suggest that the dbg kernel should not end up with > resource problems: not that much gets very far. So I'll > probably stop it and substitute the debug kernel, reboot > and try again. Still for nodbg kernel . . . The "vlruwk" processes do occasionally instead show a CPU?? . Nothing seems stuck in only one STATE. (Live lock?) As for using the dbg kernel instead . . . Most of the time that processes are showing CPU?? more progress is made in building, but basically one builder. vlruwk dooes show up, gradually showing a larger fracion of the time. ref 02 .. ref 32 are still in cpdup -i0 -o . *vnode is showing up some as well. N process looks to be stuck in just one of those. (Live lock?) The debug kernel is not reporting anything during this so far. (some time goes by) At this point vlruwk is fairly commonly what mostlt display for the cpdup's that are not finishing --but none are stuck in vlruuwk . Looks like I should try without the 2 patches (15107 and 15122). =3D=3D=3D Mark Millard marklmi at yahoo.com