From nobody Sat Aug 19 20:41:56 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RSrJh1TMSz4mSt3 for ; Sat, 19 Aug 2023 20:42:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-20.consmr.mail.gq1.yahoo.com (sonic306-20.consmr.mail.gq1.yahoo.com [98.137.68.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RSrJg0Q9cz3Jwg for ; Sat, 19 Aug 2023 20:42:10 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=Man5feW7; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692477729; bh=DbmaRmo6j39C+1MCwztFu9UUrPw4Q/QoV87Xnv4By38=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=Man5feW7Ab4cWO0EVU3/a5f9iv9uNqzj+SIm8j7tx34VVucq7jDZX0bNe/v9j0DhRRIvMy+1+sQFlBD/u8gKjMw4k420fk34f+NelSggMU7KR6TEFntk5Vaxk3gT/m4NtDZhV+9NvTtpZ/3kqKREY/v5vZDEXaKWey2t3xg+7CrfIJGngL4YB91f2qmloZc4xMmDWgs+G0AVaWlhZ6LBcKSrCthqF38MfZ+DPj7OADp7uP7q0VvUVhbRNHUOQ2or3UOp+PIuaPFM2KEihY35jbJ79MLtD0nC29GIporLA0ALO1EQzH7TwovIej3s5p3p+XLDBRpUJxwUaf4qJVUdRw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692477729; bh=VX6DL9T5W8RGe/2MLth9oKKf+BPPbSjoOsn2571x3VT=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=T5oQGwTWn7U1qKaL5eVVNhxOVz81QVlsOrV9AQHFmjjL8D/BZg8KnEu2hmnfv59Uwhf4cAFlluq7+JsDv4JlLJp3X6cQxYwumu65Utg9FlLoEpgvBtQ1lq7PK3kyeSxOl6TfBZxqW5v/KjRJnHsGeMWg+l8sNbdOatGg2xoKCLOGOgr/NuZYrkWbrZ0HaSneEMi9AvyOCP3s/1rpuT1mEeGT8dbvRAULVHe6njDaArZBu9s9dZNqTjofa4Kt5fiLD8Vp/togMAoik73JGnPYfGkW+dhz15I9Pdwq3LmjkfgI7YJeU0Rg0WvcDEX2FEAuUH4zwnBwazlkUgOqCQincg== X-YMail-OSG: CEdfHGYVM1mkajN_wjDN4nYkvS6pSitVDGrHq2F1gNC2qEI7hHbJb_xH9yHkMdz .6QvsnTIRI2zLaRe4BtAJtA1E6KKkQ23Ftb9rbibwx1FqXVHbcqOG.JR08Yz1bwHUcQqrxxKjrAU b6748AyQ_JTCpPfC0eIiwBz3INrcq9WuwgEOzGkw5tlj3kwQkcLiANduZk72y9SR9mT5lqXJSGbs 8j6TsstF1vYW7z8Ydwp5MPj_YD4J3cI2pKCUfx5xnKrqGQ7iIUaY8E9vUT9Jjq1COX_XFYL9sNVk Tfch2tJ4XzI4lS18rgQt5K2I0SLxOrUxoehagum1HWgP_i2pIIqn7nN8HURmrjx1oCzlbhvgUs8d 6gBUsSCpTKssO19RrkZl0.Ur20683HUAadAejuUiOYtsC4kjxuUWmw3dJfRB624hy2YhseazDHva PoKamL9CSZphfvomhc_Str8xJYHmVjUHvARq_Gyo8uH4muk4hn39ZZW9XW7bk0EyhsmHbV5HiKAI fyt3x35RJAOzNMRxpPKXXO.YMmDNoM7KsT7hoNSxHl9BHyg1jEYVJNA6nFfZX4ojlwNnsNIfpdv6 zlZ5Oojx9Blw75hlYPMcN419qUvrESgcLwGuOie3p1m8dg.Jf6cEP45Dvkt.2DVjVPtCQhYd5rsN eQRZQvNDb9Lj1V19LgwUz3HqDbnT_5mDVxoZhFBxjdH.iMwP4oZE1hD8cpvPfg_T6CEpBxCX5cFR pZgJ9keO6GWqhbEnszCviEgeqaE6nWPXhxiebr7kF0aqFebVzJkWvTXnkiQO3xkllUyHM5QHOfUn zV3LSIAbDMoYP6YkYw_IPsj.uniiqHQnrVIjWMIBo4rQwNMJ1v2JRGMG4ZP3rQRaJa1XJqawEVDI nYdjfjEevwZg9uOGOo8smy2QEtxyvcIGXwypJYQgPa1gYV6Bf2Lx3uD.dpr69eGzPvtifDVuc4I6 KkWS1952oY3V6_WUSHfAWtc10CTNisc_1eSrU_6pEw5JlpdWwVR5fl62wCwcFoOs2zX9PmvM3mc5 F2KBwR82o_YmI_O4iJ3yu_D7IS.m.VAf6Lpw4276vxazseCUEpavPhJ7QlvZWF_1tbMMoRaQ3g.v vqf6X_GUQEq85KqPaAhVxEk9LhV3.2inTqm1N23Q0QvIVLgnHT9xb3UVPBhUNC2DM71_swk6qV9a SB9QYDUCMMN.c5H094Ig3gTvqmYUBhH9VRm2D9FQ5bYvUKXCNHfmI9zoJVd.FI4oUOpWG6wkVa82 NUIXTPzcgbUgL6BURU1VEQOVeNpWhbDyJOXap26KW9TaRR7S7JMSFqbX0X.WpSLkInqfWh5JZh3D 51fNFMH5DJWUhl4kFr7Y2zJ7sjT0A_LrD8BUg32YUfOgQ2EUjX9pp.bXgia5dK_YkT6K89G_qPKf cMQChyUe4AidM2l9er5DYf1n8q67W_sx9sWPzLlw0aGi2xnuS3BL3G9RH1t3K1cZvIX9Y3DWb43n 47YmeXeKzqyW9K_hWWy.8Dq2rhDTFtWh5SNXFW7CGqeiVtDU01Pk9HyLIV9kXrnvXmb0N1Pk6iqB apfDcrU9LhkP_IaSKOLrDNyVS18drrp4i9gXXO90MFQWmOKCs_s2HzJBZD3hHBouQoz1jLUAoYuo x6fak94EGo88WHl1UhbEfmTCgskscb2jvxef5qFLBq5kkM5jYouVTSpaVhbYk6AbQSS1iUzXJ96I mf.P8hKaIbFaR8YccI8h03Q3i3JH2fTFxTQjYSCwGHfYRQQZNBBOEQ3IYodoRaV3syQe6VRRd6Dp _tpxiRegZTbuNQc.pTD1swR_203bT444hu4CNZZBzMEIVdQD5lS99JQ_10MTjqI.Iayi68LMHf3U fmOJddSMKQD3bJzNzyMQrugFBFFL6B4VZ.jxDVT2VcwQqUc.icnDta1_W_yJzDOxZJkJZfH5_qzg A0ANEoSzdoV1voU8r6oicxsYdiTtcfo5gTNpFDbjtWbZxT27RjjV1spdJbc1Pcxr.rbuWF9FirB0 w_1DQXZ3Tg0ymuxjXJbVdDceCP1sIeO11DZk2JpMTcypSJ2AiL2zIu0XoC86ilNXKIKHUNCgmqWF fdbnVHm9TaXmII13jUGukb7BcYWgG51v8uOQ3rfnzK.ul8vyEMShzEk3fSJq.yeDCAnIVxUajNEI uwtQUCBPgkNaDEAdFKyxo4LESLDBlr_NOqBeYkXHva8DEcl9cTRpzM2dBnhZA5EuLFgRjFVpXgRn h1K1j9hxIARTS4RO1jB07R7zf2MTQW_IWLMQ6NVcDVsniDdLA8Q6zdjrsrg0C X-Sonic-MF: X-Sonic-ID: fb761505-5ba6-44a6-b11c-3c2dc42406e2 Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Sat, 19 Aug 2023 20:42:09 +0000 Received: by hermes--production-gq1-6b7c87dcf5-j6k2s (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 369796f3a1c3d0fd00d8a854784b184f; Sat, 19 Aug 2023 20:42:07 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: ZFS deadlock in 14 Date: Sat, 19 Aug 2023 13:41:56 -0700 References: <59FCB309-4A55-4924-98C4-7ACCA70FD299@yahoo.com> <0F2C42B4-36FF-443A-A174-5B0CC57C4FC7@yahoo.com> <3AA253E3-C4F0-4AA3-9C37-D77E7527A458@yahoo.com> To: Current FreeBSD In-Reply-To: <3AA253E3-C4F0-4AA3-9C37-D77E7527A458@yahoo.com> Message-Id: X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.83:from]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.83:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RSrJg0Q9cz3Jwg [I forgot to adjust USE_TMPFS for the purpose of the test. So I'll later be starting over.] On Aug 19, 2023, at 12:18, Mark Millard wrote: > On Aug 19, 2023, at 11:40, Mark Millard wrote: >=20 >> We will see how long the following high load average bulk -a >> configuration survives a build attempt, using a non-debug kernel >> for this test. >>=20 >> I've applied: >>=20 >> # fetch -o- https://github.com/openzfs/zfs/pull/15107.patch | git -C = /usr/main-src/ am --dir=3Dsys/contrib/openzfs >> - 13 kB 900 = kBps 00s >> Applying: Remove fastwrite mechanism. >>=20 >> # fetch -o- https://github.com/openzfs/zfs/pull/15122.patch | git -C = /usr/main-src/ am --dir=3Dsys/contrib/openzfs >> - 45 kB 1488 = kBps 00s >> Applying: ZIL: Second attempt to reduce scope of zl_issuer_lock. >>=20 >> on a ThreadRipper 1950X (32 hardware threads) that is at >> main 6b405053c997: >>=20 >> Thu, 10 Aug 2023 >> . . . >> =E2=80=A2 git: cd25b0f740f8 - main - zfs: cherry-pick fix from = openzfs Martin Matuska=20 >> =E2=80=A2 git: 28d2e3b5dedf - main - zfs: cherry-pick fix from = openzfs Martin Matuska >> . . . >> =E2=80=A2 git: 6b405053c997 - main - OpenSSL: clean up botched = merges in OpenSSL 3.0.9 import Jung-uk Kim >>=20 >> So it is based on starting with the 2 cherry-pick's as >> well. >>=20 >> The ThreadRipper 1950X boots from a bectl BE and >> that zfs media is all that is in use here. >>=20 >> I've setting up to test starting a bulk -a using >> ALLOW_MAKE_JOBS=3Dyes along with allowing 32 builders. >> This so 32*32 or so potentially for load average(s) >> at times. There is 128 GiBytes of RAM and: >>=20 >> # swapinfo >> Device 1K-blocks Used Avail Capacity >> /dev/gpt/OptBswp480 503316480 0 503316480 0% >>=20 >> I'm not so sure that such a high load average bulk -a >> is reasonable for a debug kernel build: unsure of >> resource usage for such and if everything could be >> tracked as needed. So I'm testing a non-debug build >> for now. >>=20 >> I have built the kernels (nodbg and dbg), installed >> the nodbg kernel, rebooted, and started: >>=20 >> # poudriere bulk -jmain-amd64-bulk_a -a >> . . . >> [00:01:22] Building 34042 packages using up to 32 builders >> . . . >>=20 >> The ports tree is from back in mid-July. >>=20 >> I have a patched up top that records and reports >> various MaxObs???? figures (Maximum Observed). It >> was recetnly reporting: >>=20 >> . . .; load averages: 119.56, 106.79, 71.54 MaxObs: 184.08, 112.10, = 71.54 >> 1459 threads: . . ., 273 MaxObsRunning >> . . . >> Mem: . . ., 61066Mi MaxObsActive, 10277Mi MaxObsWired, 71371Mi = MaxObs(Act+Wir+Lndry) >> . . . >> Swap: . . ., 61094Mi MaxObs(Act+Lndry+SwapUsed), 71371Mi = MaxObs(Act+Wir+Lndry+SwapUsed) >=20 > Status report at about 1 hr in: >=20 > [main-amd64-bulk_a-default] [2023-08-19_11h04m26s] [parallel_build:] = Queued: 34435 Built: 1929 Failed: 9 Skipped: 2569 Ignored: 358 = Fetched: 0 Tobuild: 29570 Time: 00:59:59 >=20 > Not hung up yet. >=20 > =46rom about 10 minutes after that: >=20 > . . . load averages: 205.56, 181.58, 153.68 MaxObs: 213.78, 182.26, = 153.68 > 1704 threads: . . ., 311 MaxObsRunning > . . . > Mem: . . ., 100250Mi MaxObsActive, 16857Mi MaxObsWired, 124879Mi = MaxObs(Act+Wir+Lndry) > . . . > Swap: . . . 5994Mi MaxObsUsed, 116589Mi MaxObs(Act+Lndry+SwapUsed), = 127354Mi MaxObs(Act+Wir+Lndry+SwapUsed) Just relized that I'd forgotten to reconfigure the USE_TMPFS=3Dall to be USE_TMPFS=3Dno so what I've done so far is not a great test. I'll still probably let it reach 3hr and get the summary information before I stop it, adjust USE_TMPFS, and start over from scratch. =3D=3D=3D Mark Millard marklmi at yahoo.com