From nobody Thu Apr 21 12:50:42 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 27FD61997B74 for ; Thu, 21 Apr 2022 12:50:51 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [IPv6:2a00:1450:4864:20::12f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kkcpf0L0Rz4RHB for ; Thu, 21 Apr 2022 12:50:50 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-lf1-x12f.google.com with SMTP id w1so8522443lfa.4 for ; Thu, 21 Apr 2022 05:50:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=pQhFeqih9jjYAgNMe0HJ0GyXfNt+0KwauZfsyVqCgp8=; b=deU6LR0/NDHhl6tQhLf/EBc5L8X1B6+LT50tCu09xpu/a6KJArEzTXTSOp5mVk0Tz2 rA0UCOh1F91r0Jac/3XbqdK6VwRXwIj4yOpmTZPWTDcYM/j40bKCVIERDJ5yJq/6J7A9 U/6yM56Hz0/RVfOt5WEllfUqZk76Cou6Gdl+1pEKQKY0sdmG87QC9n2uHluT6FGGnYz9 Do4MjmZLeLlBNAYC/2r1zD4g5bBnPASVmBFYINlB05W6ectRkL1Fh6jPA+V+QVOu5Gej Dt2uPb5gZuUC/TnzhggpGzTeeURBeIZfi0awepBaTgs4jq254QrWCo591eQJeeZj4vMa 603A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=pQhFeqih9jjYAgNMe0HJ0GyXfNt+0KwauZfsyVqCgp8=; b=hf6EFErzXsH5JIGL8RElwCRZ4WVycjE1LVPB7YtPTGG7T4r3RkFb/iIcQcpaUdUhlB rsUVMzezHp3JqT3E5rb7tIPlDXb18vMOQrjrICdMjnIaNSgUh/jns8GGB0ED0mr+Yq9P 2UH61X6xDZ+ILFMqTNqrinECy9dt/TEhl7drNPZIK5CgmMlEh4ueJilFjou42N3U3m9x 6y+7Ei8tPXZxZ83MMrIWBTMJlhyRN1nqCoXfsFUSzoghyAlUEPAW7rR+350GLBJxn71L c01lOQMQa+i3SKnGoJJukZISnRJheISRiB7z+r4KtdLvPIkXLLQKuu/gbAL6Z4F2Z6A2 P7pQ== X-Gm-Message-State: AOAM532eW10NlnoBN3Zbc5y/q5OupVs4j9Pf8qPPdQiXD563L4n7EozI ZLiN9oW+OhxgFT5sygo/Mux/o4ukWL88Yoev9NmRO+yo X-Google-Smtp-Source: ABdhPJyzlG/KVvzFlOKDz9CekG/z8G64mw30egFlb3/0Aq3p4R9m07sIx1UnpanbCzQvjlbaYT1S4fbuc7I75NmReWs= X-Received: by 2002:a05:6512:10c5:b0:471:a703:bca4 with SMTP id k5-20020a05651210c500b00471a703bca4mr9266847lfg.581.1650545443229; Thu, 21 Apr 2022 05:50:43 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Received: by 2002:a05:6520:6145:b0:1bb:7433:4cdd with HTTP; Thu, 21 Apr 2022 05:50:42 -0700 (PDT) In-Reply-To: <20220421083310.Horde.r7YT8777_AvGU_6GO1cC90G@webmail.leidinger.net> References: <20220420113944.Horde.5qBL80-ikDLIWDIFVJ4VgzX@webmail.leidinger.net> <20220421083310.Horde.r7YT8777_AvGU_6GO1cC90G@webmail.leidinger.net> From: Mateusz Guzik Date: Thu, 21 Apr 2022 14:50:42 +0200 Message-ID: Subject: Re: nullfs and ZFS issues To: Alexander Leidinger Cc: Doug Ambrisko , freebsd-current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4Kkcpf0L0Rz4RHB X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b="deU6LR0/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2a00:1450:4864:20::12f as permitted sender) smtp.mailfrom=mjguzik@gmail.com X-Spamd-Result: default: False [-1.94 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; NEURAL_HAM_LONG(-0.93)[-0.926]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_MEDIUM(0.99)[0.986]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::12f:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-current]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N On 4/21/22, Alexander Leidinger wrote: > Quoting Doug Ambrisko (from Wed, 20 Apr 2022 > 09:20:33 -0700): > >> On Wed, Apr 20, 2022 at 11:39:44AM +0200, Alexander Leidinger wrote: >> | Quoting Doug Ambrisko (from Mon, 18 Apr 2022 >> | 16:32:38 -0700): >> | >> | > With nullfs, nocache and settings max vnodes to a low number I can >> | >> | Where is nocache documented? I don't see it in mount_nullfs(8), >> | mount(8) or nullfs(5). >> >> I didn't find it but it is in: >> src/sys/fs/nullfs/null_vfsops.c: if (vfs_getopt(mp->mnt_optnew, >> "nocache", NULL, NULL) == 0 || >> >> Also some file systems disable it via MNTK_NULL_NOCACHE > > Does the attached diff look ok? > >> | I tried a nullfs mount with nocache and it doesn't show up in the >> | output of "mount". >> >> Yep, I saw that as well. I could tell by dropping into ddb and then >> do a show mount on the FS and look at the count. That is why I added >> the vnode count to mount -v so I could see the usage without dropping >> into ddb. > > I tried nocache on a system with a lot of jails which use nullfs, > which showed very slow behavior in the daily periodic runs (12h runs > in the night after boot, 24h or more in subsequent nights). Now the > first nightly run after boot was finished after 4h. > > What is the benefit of not disabling the cache in nullfs? I would > expect zfs (or ufs) to cache the (meta)data anyway. > does the poor performance show up with https://people.freebsd.org/~mjg/vnlru_free_pick.diff ? if the long runs are still there, can you get some profiling from it? sysctl -a before and after would be a start. My guess is that you are the vnode limit and bumping into the 1 second sleep. -- Mateusz Guzik