From nobody Sat Aug 19 22:41:04 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RStyC6fWGz4qK4j for ; Sat, 19 Aug 2023 22:41:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-24.consmr.mail.gq1.yahoo.com (sonic312-24.consmr.mail.gq1.yahoo.com [98.137.69.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RStyB5YvMz3Spj for ; Sat, 19 Aug 2023 22:41:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="VBh3/DFG"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692484880; bh=rmnu/zQViM1luMQlR/MUpxLw72I+KTFc5/eKSVA8wwo=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=VBh3/DFGT5IHVuOVMjAk4Q3h6osfMRKVHBFfAuf+rjY2jjGEJL3PozMnH4WdLC0ZktZ6KBsznohrfc7EuuvpWTqUh03Ay5zTYNGmmfVse1mo6S4W4sJAIFWP0MTXxcOeRoEV09ygJqLKGkTKmP87TYGxgmQhWMo2Z2linu9nT5APU7Uog2bE9475TPyjE6BITnR9hYWpgw0DrZ/r/PQIjWNEOn8NpyhwymMIZM/jYxOnxJ7Ytfx8HjRMIIaFBjpYqBRHVXoLVMbdamZKPdVaNQiFyanATFivYp3ChaU/1U78tjT3KHSb/XDefS6wTsbGP3Br6/QrqkvQksT9eNbcOw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692484880; bh=AfZy4r/SkCtPKum5giswd3ueagdCSBJ1Z9Ovgdcm72F=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Bzv2yFZuiQbZ/2xfmeSgGzjqS/skDfVkAi4E89rYr0+PY2klEYRRsA0H/pa0Q8APaBXZ079f5UQ9MppsRsSGgRuKcvZamiRHTA6xseIS1r3/IeldtOmha5+ydV9nxHRctSf1vS3RibjvG8WNlSOgD70kmVrdUa8FCea9z4rUctai86VNkn+JNXLRIJuVaxVScP5xpk+SGQ9V1E89efU+9w7AF3pLex5StTx/tZAznqKbOdNkf2Aonhd618BzZvon9BM92LkiiSQk4qn5Gri6fstzCEdpuC+kct1xlLpzBAhhNJ3KaEuAUFNHZ8QmGXnonGv5Uc6kn5itVlhWF6dLqA== X-YMail-OSG: t9TvUH0VM1ngOhDDGBgu.3bPw0ljH6K2F1_J4rvN0rmVNZU44b5DSkP_g_EfOl_ ZYRRiPnIwvuRBIWCq99iUbksHaxlOFk6nfJO7aFhNM35XjPKZEcB9IFQ9e1EnaTK2xLxoFuTw15n 9QHhkmCSNJKnkXkVElwtxnd0HQTYXgbRtenD5bYcMTs5or9sdBQF9wmlhIoFbxB3_ntO.si29BfB DlFeyQWzKRhNOrP_aRxTc_2_gKLs32VHwtZmUehU7_NHIKWJJinNC554O4DRx26HBxCu2qx8AGj. bzQI65JN9BKHlJw1UGrawEw.BjYb9fMhQLUHF15SURx_cVHVlv1FRrT6jkj_XzukAOdmzU1vrsQS JXMDFgZZqToaQGjzZQmPjMvEX5923rGu28VSCtMGbXq.PxOI3RGIZpVjFw8L9dChdU_elzQlUXpO 5_bPV2lrCdhYE3o3KnhDVa0VGGfJHzm4yYJbXZw415ugqFBDt71gAQIzXC9dSHXbp1KXRrBJuABx SsXWabrVAzA650HfqpnxmBAUpX9JFLowzVop_lNNEBLUPpxRjwPhjKyoX5dtIt0PyhM6Tt_w1LNz I56qtk7s8ZE07oBTHD4zlk4R.CYuOZp77sYz5tRXoRp_qsJoje31QcYaMW7Ikna4IpvW6gb9HLLR RoMheIisELGGTe8XqXRqaD5H9aAxoon5KeCurpvQlpx4rbzn4ag5_OXAS8tRHmleYcIV0ZOpe9eW 9n7Z90vk8pLO_Mflpa7iJ7avqGGrWFlmDynfazPCWU6.hklgSts3NGiQTEScUEPmJSG9KzMlyNyv ZwX7Z0Zjbvzgd6s_c91FA9X8pI4E4I42yH5TRjEFYl8DjAGG78ADKCaGN2Kjc0XYjk1OeY8UTpP6 vxLaLmgB9rGXpzj4NVaFfov7YEtRUrFN582pivNRWls04yLUp0zI3uTJcRfWBtQPJEq5I.4IrRkD wRajTJ4Eh6deYqxWFdfA3NKLXnZl9vDxTtieDj8CqM_1nbfR7V1vRlMzCovU6SfkcuR5Vx.p7LOv wC.kfjKopaDbkTdNposq_qboAg9XaONt7qjv1Dy9TDzjaVgWTyVXEUAMmTOcQvJ0F.W3_neS5S_P hmHmHW9vJG_eA66nkL97yS3efN.sAe5n2mhcKBdS3qdW2YC6JGESbPOxP63op25ivX.SrX5SinIG fpmRJc0DUhPYi7Q7dclPSf2QPgJZLhOGfFNkezKMNbGawcBj2BAe5KZ4vUlPhuGDOkS.DQ.VfJR3 YshcXGf.WK6mIOJxaZ4OLW5Cr9XWkThKgSceaUO_QXOzmQb6hrFTomoR7CCso2LfretqT2nd2KTZ Y3eBRxRaEz30cU6q6hmYmoQEbTvQtXvyQBJbCZ1OcnjVpHI0Qjh0rVrovSnrgp0Hn6KSgg24j7Zw AYqz6O4pGfsT1lKDP5BOHa.rpAKP6YdBfqyP7cIrA8JHNoCWWB.rJCGlYhg77mZcawVd079QYkpu A6bUEDfu64MkMWsLU4WztrF5DUFtV4rvcxenm0ApAGd4SaXik4UjnRcJhm3wmQ5PZV1eyvQI5Lop zQl5F6t5GOz8xi.XsrirGbht3SKGEdNtXs1i8tbVsTiXtu1ro2MkfQv8AKsXXM006aza9bIww34a xHuj_Aq9VMOv2Osdb95x2HmqCHtLIgLx6BFo4nXaIAHXtCwCw2sTXqWD9..3c6bBwkyb6iqMcLhm GivjEBS_XorZZugvr1WcHGtPOVhDceI5RhIvI2e_vKFiXSCNTFN6jl3aD1_dV49aioVzoG1XN.iA exOdmwMDsTy2gAbKKzZdoSfdN22zXYGP.7WZ_.C5_va3X31RpL1zg.iS7vBawxXxLZxYBPLBafRS KV9YVv3IssmqMhGgksG7ciIsCQ4mJgUtJazbjkDa0AgJycFEQofMTRKGbl.SQMDF4YgcqqjKJJ8y FQbZS77b0pH6OR5HqzaNXJhRwwgowPZ7lKFBUAtiisbsmA2LYaE_8b0Hs3hmWttMVmXIKn8E75g7 s1KJracluXYRXm5uZds6DQ_NttD.VW8OV6RxD.VKpT8zr6m.f1bBGFfcyPYysl.AsTKprggSbogV BtoRPb9J4rIxGXpJimTIL9FmfunC6xLauhn2x98myFAgU_cUkt_GtboMVr3SQBkj4kTbQ6NyRtyn 5AdifR9.4W10nNSfVLsl9ULxnab_44FIVh_leez45.hhGv5qk2kIMSsHe7d7cVJwTnRa7OudSwC6 qPROTLneXLjY9TDuuTd2RPs7h5ULLrduAlC9WgzXUvsm1oE6qxkAsaVVkvhITWw-- X-Sonic-MF: X-Sonic-ID: e4c11f37-bb74-49b1-8b52-9844073b8b85 Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Sat, 19 Aug 2023 22:41:20 +0000 Received: by hermes--production-bf1-865889d799-r6v2w (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 91cdd64db58aad3a703eaa54ba53c5b0; Sat, 19 Aug 2023 22:41:16 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS deadlock in 14 [USE_TMPFS=no poudriere messed up from the start, lots of "vlruwk"] Date: Sat, 19 Aug 2023 15:41:04 -0700 References: <59FCB309-4A55-4924-98C4-7ACCA70FD299@yahoo.com> <0F2C42B4-36FF-443A-A174-5B0CC57C4FC7@yahoo.com> <3AA253E3-C4F0-4AA3-9C37-D77E7527A458@yahoo.com> To: Current FreeBSD In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; BLOCKLISTDE_FAIL(0.00)[98.137.69.205:server fail]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.205:from]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.205:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RStyB5YvMz3Spj On Aug 19, 2023, at 13:41, Mark Millard wrote: > [I forgot to adjust USE_TMPFS for the purpose of the test. > So I'll later be starting over.] >=20 > . . . I finally got around to starting a from-scratch bulk -a again (based on USE_TMPFS=3Dno this time). This is with 15107.patch and 15122.patch applied. This is a non-debug kernel experiment. Interstingly it got: [00:01:34] [01] [00:00:00] Builder starting [00:01:57] [01] [00:00:23] Builder started [00:01:57] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.4 [00:03:09] [01] [00:01:12] Finished ports-mgmt/pkg | pkg-1.20.4: Success [00:03:21] [01] [00:00:00] Building print/indexinfo | indexinfo-0.3.1 [00:03:21] [02] [00:00:00] Builder starting [00:03:21] [03] [00:00:00] Builder starting [00:03:21] [04] [00:00:00] Builder starting [00:03:21] [05] [00:00:00] Builder starting [00:03:21] [06] [00:00:00] Builder starting [00:03:21] [07] [00:00:00] Builder starting [00:03:22] [08] [00:00:00] Builder starting [00:03:22] [09] [00:00:00] Builder starting [00:03:22] [10] [00:00:00] Builder starting [00:03:22] [11] [00:00:00] Builder starting [00:03:22] [12] [00:00:00] Builder starting [00:03:22] [13] [00:00:00] Builder starting [00:03:22] [14] [00:00:00] Builder starting [00:03:22] [15] [00:00:00] Builder starting [00:03:22] [16] [00:00:00] Builder starting [00:03:22] [17] [00:00:00] Builder starting [00:03:22] [18] [00:00:00] Builder starting [00:03:22] [19] [00:00:00] Builder starting [00:03:22] [20] [00:00:00] Builder starting [00:03:22] [21] [00:00:00] Builder starting [00:03:22] [22] [00:00:00] Builder starting [00:03:22] [23] [00:00:00] Builder starting [00:03:22] [24] [00:00:00] Builder starting [00:03:22] [25] [00:00:00] Builder starting [00:03:22] [26] [00:00:00] Builder starting [00:03:22] [27] [00:00:00] Builder starting [00:03:22] [28] [00:00:00] Builder starting [00:03:22] [29] [00:00:00] Builder starting [00:03:22] [30] [00:00:00] Builder starting [00:03:22] [31] [00:00:00] Builder starting [00:03:22] [32] [00:00:00] Builder starting [00:03:30] [01] [00:00:09] Finished print/indexinfo | indexinfo-0.3.1: = Success [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 and is still that way minutes later. ^T shows: [00:03:31] [01] [00:00:00] Building devel/gettext-runtime | = gettext-runtime-0.22 load: 13.02 cmd: sh 2187 [vlruwk] 570.19r 0.62u 38.60s 9% 3948k #0 0xffffffff80b7701b at mi_switch+0xbb #1 0xffffffff80bc976f at sleepq_timedwait+0x2f #2 0xffffffff80b76770 at _sleep+0x1d0 #3 0xffffffff80c5b435 at vn_alloc_hard+0x2a5 #4 0xffffffff80c50b72 at getnewvnode_reserve+0x92 #5 0xffffffff829b9b12 at zfs_zget+0x22 #6 0xffffffff829a6a8d at zfs_dirent_lookup+0x16d #7 0xffffffff829a6b5f at zfs_dirlook+0x7f #8 0xffffffff829b6410 at zfs_lookup+0x350 #9 0xffffffff829b182a at zfs_freebsd_cachedlookup+0x6a #10 0xffffffff80c36a0d at vfs_cache_lookup+0xad #11 0xffffffff80c45141 at vfs_lookup+0x581 #12 0xffffffff80c44238 at namei+0x238 #13 0xffffffff80c63b5e at kern_statat+0xee #14 0xffffffff80c64237 at sys_fstatat+0x27 #15 0xffffffff81049a79 at amd64_syscall+0x109 #16 0xffffffff8101f11b at fast_syscall_common+0xf8 [main-amd64-bulk_a-default] [2023-08-19_15h14m10s] [parallel_build:] = Queued: 34435 Built: 2 Failed: 0 Skipped: 35 Ignored: 358 = Fetched: 0 Tobuild: 34040 Time: 00:10:52 ID TOTAL ORIGIN PKGNAME PHASE PHASE = TMPFS CPU% MEM% [01] 00:07:29 devel/gettext-runtime | gettext-runtime-0.22 build = 00:06:32 25.4% 0% [00:11:25] Logs: = /usr/local/poudriere/data/logs/bulk/main-amd64-bulk_a-default/2023-08-19_1= 5h14m10s Note the 3:31->11:25 . Top is showing lots of "vlruwk". For example: 362 0 root 40 0 27076Ki 13776Ki CPU19 19 4:23 = 0.00% cpdup -i0 -o ref 32 349 0 root 53 0 27076Ki 13776Ki vlruwk 22 4:20 = 0.01% cpdup -i0 -o ref 31 328 0 root 68 0 27076Ki 13804Ki vlruwk 8 4:30 = 0.01% cpdup -i0 -o ref 30 304 0 root 37 0 27076Ki 13792Ki vlruwk 6 4:18 = 0.01% cpdup -i0 -o ref 29 282 0 root 42 0 33220Ki 13956Ki vlruwk 8 4:33 = 0.01% cpdup -i0 -o ref 28 242 0 root 56 0 27076Ki 13796Ki vlruwk 4 4:28 = 0.00% cpdup -i0 -o ref 27 =20 In other words, it is messed up from the start, not just later. It does suggest that the dbg kernel should not end up with resource problems: not that much gets very far. So I'll probably stop it and substitute the debug kernel, reboot and try again. =3D=3D=3D Mark Millard marklmi at yahoo.com