From nobody Thu Aug 24 07:22:00 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RWZKQ5fYrz4qlLV for ; Thu, 24 Aug 2023 07:22:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-21.consmr.mail.gq1.yahoo.com (sonic314-21.consmr.mail.gq1.yahoo.com [98.137.69.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RWZKQ3Jzrz4btp for ; Thu, 24 Aug 2023 07:22:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692861736; bh=Ecs5TJbzYhfH7iOmtuTwm1Phwv1tkjRCyBuNaOxrBVE=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=P8T+K9myq87WD1/bqEUpgtzdyK0DfHGlsDzoRKjlBYJj5wR1f3eL1Ht+/OXMVB7foJ3s8RJsnf41hcvNbLJAHcuppwC/foE/6rxikkwMS2c4cM2Goc2LtNsZ2ma73QZk0OgUrFs/AUZXkrNbIyweryF+6nDDWVmX4tpp2O3nF4btMMu8ZaugyaiYQl7sBmXlkokGyOK0+fc9in39F0Aq8PBl8cI4R5sNPKeBcnpCGg0FS0bAK45r29wr+Of7aYHdmihY8aFGZRJVcdpEwhhZeJ94lnwXADpuEOL4f99IFlv+laBVolZsi+k5m81d/rA0QuS9OoLqJFysCBW+aY7M/w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1692861736; bh=2sCIogA+PhHTe4buioRG11YKjTDOTrNw9t2GPGjlEUM=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=rrcjYJPRreJV9HekrdXH3eT3+f6hQ5920EcuFAQKdDrW+Ehliz5CtpJRAIboeQ7BW8AsQqA2DAKGPiEreEkwVWYZiEEezKW/q6ftmrHhtqxkPY7REoyXqppnGVG9YRLG+M+827EL647kGX313zj2Ed3btxx+DEWEeK1/1+M8p/p5zJRkLkrliTtNDUx5rmq12YIGLJXjLPWVDoCJk2nmfX3nih5BhdUqB/dEYxQHYPPNzBl8pO+Qj4MgRppdLONUY0HS7NsZIX1JqiqAYjbvN/JD6ak/kJzCBeMLowYC+4lDWXAYg7RBEkJXd48wdUyZyhiUTJP1wmm0xX6UjQ9pLg== X-YMail-OSG: BLppw.sVM1mzKhs0.17oslBDLwMhnFNDiyvb0ymKDMFMIVFBgUAko2H2ebtLReC c65c22y6h893SL9SgrTSJVKlICBuqGL9d2CYPEScycBih68Ei6DFR2lwtYDJ2Y5JIGLjg0c.vvLO ULRAN.cyB2jIDHU3_pjuWtQtTpeh1FotxDodV7E6_4DRfNJYMR6m8JNS6nQ7N3a6Web4WAduQ81_ lhFZ8rzTSv2DooCjLykZBvN0e_DwaAm1qs.T5sOyx5.GmFxpN0vwBvcCF8gCON.hvBgOYt7.E_r1 yZK0vCwLnsurzDH8ejve0Lh947wEsrXgj9patKi7z71JKdvzzGaEsikEiFBYfixMK5_rf7waKN2Q uBKKXh35lwop8b6ESs7C3h1fs_WhxGJ.7phA0Nngz.M454Htso_99gLb4aiuR8lf0BpydROLrdwA 23Syv6Iwpcq8H8BANs.cFm_K0mMpO7LbrkFMr0gchXtN03RywNTqtQuyAgEYmq1z.DwiAtypXyAD h06ol7Tf0nICsX_wJXGSw11iIwdVSYaRq3k.Rcs_F48dvWHPeBJJMV7dZ1TQQB2tPf72TmboyGaj QsCQWH1TK3n8FfcBdDeT3N.h31xrne92faxZiNX11XAZyNta_81LPXoOAqSMxrdRQBDIn0qmq.M6 VFGeJ5gV4_3mQe7l4cE681dpt7frDPLnFZDVl0EHZ4zAp09xoFX3B8W3Kga_VyBr.JaJQsLbkZJR nR66ZdIZC9KfSyY7M3B2XzkgwQuRdkQelQxt_lOA2omigTOcltOYveSkTZklCy.88Qn8Sau5JdIL RFr9Un6X_CW1A21tMczTuuBVybvZ2gfGklDz9wV8ng99DNW_zksHWmsvcENhFI00b0vdDWreCDKP 2dw6ay0H.6yrysNkunXdf.2zAcR7uZMX833xV7.O3uMGLJz6pHjlh1hVnjlXbjRBycwgeiuCtblv qPXkoWTSFJq3SCIZ_hXqtW.AVO35csPY3SF5spvx.pR6wgUBXX7dXFF_0fGhJgVW0C.ckk9oKgAJ 5W765si.usyD1mTc_AXeokgAKoRoeUXkL6GyDZxKNZV1oI6PJ9Jjvn3yDNqqR5eMUsmYjp_ucmqC uGa9_U68Rn6VSiMxSmQjeafYj87bNeJopUl6.fGbJPJq7QqNMftS6H4XBY1Wqqrutg0Vv3Wn1tmc DX.mTAEFXjVYo8OcV.IjBJsu94ehAzX5O7fowg4JGDnTX2UX3S7h6NdSFWmCKUUDUA42G3rVqWRE PigVCnTUMTjB8CJ1NTUFrh5sw.xowVa1FLXnwwzFySZx7oObFYWLIpCkG9sxmkqFj0BQl5l4bTgU MsI2_RL7dknV6ZG79IBR88oq6ioKojGHVUacoBsoT3bLmKZOnJJKWI7KauW_Ocx1PSqyNTPtlN.V RkVzgO8VBCbLPWstRwzqTgceT7CEWm5Zk3DopJ00Qnp.1onmyw.F12x9tudUM6sBRRmNlnUaWEt6 EpwNdLH3F0zID7NfsiuM1y6wGewDFmhoB4Yk2eKC5rBZXIbLhb6DVQDT.MfjFxvCNWKh3s8XWsub VhNBSg0rs7q9ja9ui4kN7WKYL8af5wy2.zP6yDyi8vynH4AdyXVGhNkkJW2auba6BFOg74YWuDZ_ kPwviYXgXWQEQ6SPwYcAQjA2GbSjxYj306i0WAUlKe.R2T4F.50W3234XnKlanEOklR1AKC6R7JA whWb4I1M_lzO.J9dLlPBxp_QQnHPvBGfc7PVdDTUKbGHOA5J4RpY2qeEetH3bntCW_LEYcptC.yD CRNPSrUOpCdeNJIhp8so6ycZC9Kw3J71NGAxoj_7Qqt_FDVz7FVJv._roVMJy119I.yaIT6Bva0r OMshJqey.HBOOq68kW3fJyxW5.XKsBJOY3Mwd4qCJQStSJnT.cqMFzZhrx_2n27z7Bru7twramtt sRJZJnqI_EvfmK2RwEwFF6imcXrCMZbYvyKXsJeNiInobASCTiQglIXHFj.9i8V8QCfjZYCFkgeC ivsEvIzPYTzv0c.g6maJkQHDeTUngTTOdWkG7Y33HDoFf4vHPB.udQC3U65lipA47pPvQqPjhOrE OvJsiDyMgEcMrcZxEVnd9pVLjTdb5rN8o596wTz_vseM0.On9W_cQFMdynEs0CfiY6WxQo_L_7b5 yAX227OyiU.qoiY1Se5ScBmMCqMr36vLR.Kt8N.g4griN5OMsxhwEAkbT2BSObWHwyvnC.eiS_Kz K3KP.23NnUcnlH_CSv_8QLqZmj1uKWojHBhwVce1udyRyq8GDatG_pC2C6R6OtpHH6BOCXkiXggQ D X-Sonic-MF: X-Sonic-ID: 84613e97-1997-46da-9159-6080a0c903df Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Thu, 24 Aug 2023 07:22:16 +0000 Received: by hermes--production-gq1-6b7c87dcf5-792fm (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID f4efcc3bd0b0f3a50c7492784dcb57cf; Thu, 24 Aug 2023 07:22:11 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: poudriere bulk with ZFS and USE_TMPFS=no on main [14-ALPHA2 based]: extensive vlruwk for cpdup's on new builders after pkg builds in first builder From: Mark Millard In-Reply-To: Date: Thu, 24 Aug 2023 00:22:00 -0700 Cc: Current FreeBSD Content-Transfer-Encoding: 7bit Message-Id: <2CF7E1B7-3026-4485-B2F1-3D464CF0FE4F@yahoo.com> References: <4FFAE432-21FE-4462-9162-9CC30A5D470A.ref@yahoo.com> <4FFAE432-21FE-4462-9162-9CC30A5D470A@yahoo.com> <5D23E6BE-A25C-4190-BB2C-A2D3511ABD90@yahoo.com> To: Mateusz Guzik X-Mailer: Apple Mail (2.3731.700.6) X-Rspamd-Queue-Id: 4RWZKQ3Jzrz4btp X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] On Aug 23, 2023, at 22:54, Mateusz Guzik wrote: > On 8/24/23, Mark Millard wrote: >> On Aug 23, 2023, at 15:10, Mateusz Guzik wrote: >> >>> On 8/23/23, Mark Millard wrote: >>>> [Forked off the ZFS deadlock 14 discussion, per feedback.] >>>> . . . >>> >>> This is a known problem, but it is unclear if you should be running >>> into it in this setup. >> >> The changed fixed the issue: so I do run into the the issue >> for this setup. See below. >> >>> Can you try again but this time *revert* >>> 138a5dafba312ff39ce0eefdbe34de95519e600d, like so: >>> git revert 138a5dafba312ff39ce0eefdbe34de95519e600d >>> >>> may want to switch to a different branch first, for example: git >>> checkout -b vfstesting >> >> # git -C /usr/main-src/ diff sys/kern/vfs_subr.c >> diff --git a/sys/kern/vfs_subr.c b/sys/kern/vfs_subr.c >> index 0f3f00abfd4a..5dff556ac258 100644 >> --- a/sys/kern/vfs_subr.c >> +++ b/sys/kern/vfs_subr.c >> @@ -3528,25 +3528,17 @@ vdbatch_process(struct vdbatch *vd) >> MPASS(curthread->td_pinned > 0); >> MPASS(vd->index == VDBATCH_SIZE); >> + mtx_lock(&vnode_list_mtx); >> critical_enter(); >> - if (mtx_trylock(&vnode_list_mtx)) { >> - for (i = 0; i < VDBATCH_SIZE; i++) { >> - vp = vd->tab[i]; >> - vd->tab[i] = NULL; >> - TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); >> - TAILQ_INSERT_TAIL(&vnode_list, vp, v_vnodelist); >> - MPASS(vp->v_dbatchcpu != NOCPU); >> - vp->v_dbatchcpu = NOCPU; >> - } >> - mtx_unlock(&vnode_list_mtx); >> - } else { >> - for (i = 0; i < VDBATCH_SIZE; i++) { >> - vp = vd->tab[i]; >> - vd->tab[i] = NULL; >> - MPASS(vp->v_dbatchcpu != NOCPU); >> - vp->v_dbatchcpu = NOCPU; >> - } >> + for (i = 0; i < VDBATCH_SIZE; i++) { >> + vp = vd->tab[i]; >> + TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); >> + TAILQ_INSERT_TAIL(&vnode_list, vp, v_vnodelist); >> + MPASS(vp->v_dbatchcpu != NOCPU); >> + vp->v_dbatchcpu = NOCPU; >> } >> + mtx_unlock(&vnode_list_mtx); >> + bzero(vd->tab, sizeof(vd->tab)); >> vd->index = 0; >> critical_exit(); >> } >> >> Still with: >> >> # grep USE_TMPFS= /usr/local/etc/poudriere.conf >> # EXAMPLE: USE_TMPFS="wrkdir data" >> #USE_TMPFS=all >> #USE_TMPFS="data" >> USE_TMPFS=no >> >> >> That allowed the other builders to eventually reach "Builder started" >> and later activity, [00:05:50] [27] [00:02:29] Builder started >> being the first non-[01] to do so, no vlruwk's observed in what >> I saw in top: >> >> . . . >> >> Now testing for the zfs deadlock issue should be possible for >> this setup. >> > > Thanks for testing, I wrote a fix: > > https://people.freebsd.org/~mjg/vfs-recycle-fix.diff > > Applies to *stock* kernel (as in without the revert). I'm going to leave the deadlock test running for when I sleep tonight. So it is going to be a while before I get to testing this. $ work will likely happen first as well. (No deadlock observed yet, by the way. 6+ hrs and 3000+ ports built so far.) I can easily restore the sys/kern/vfs_subr.c to then do normal 14.0-ALPHA2-ish based patching with: so not a problem. Thanks. === Mark Millard marklmi at yahoo.com