From nobody Fri Nov 12 19:52:52 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BA10A185A147 for ; Fri, 12 Nov 2021 19:53:04 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-vk1-xa32.google.com (mail-vk1-xa32.google.com [IPv6:2607:f8b0:4864:20::a32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HrTlh4bg0z3qMZ for ; Fri, 12 Nov 2021 19:53:04 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-vk1-xa32.google.com with SMTP id bc10so5719930vkb.1 for ; Fri, 12 Nov 2021 11:53:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8A4UEJDhUSBySq3W4OssQpyAn/HvTVRLW0A+ROqVazQ=; b=Ad2LsUyzu3yx3PAbiathpAt3Qurn5lVBPdqBdb+3E0MmRGgN6XzanU2Eautg0/k1tx xL5KEXCwsQ4djC2vcT38z8DV9UUrcqpwV+hkm9AbfAf3+eytaqaGlofHTAQvyl+xqJwR LErNNSoUj3I4shzWr8tllOJJV3HQlVSpexQjhmbI1sXYZZkfQz6K5T8NTZMfs00CoUUj nAfUVAIT/aNfftDQSJOzECfoEuYieHPGtrNpzBCefF8w35Y7oaTv+OXCC9Sv3ac14BrL 9gD9ycX6oNI153m0ijNcrqRZBUS+lZZY2wDwaTPg7c6u5MEKKV6PKbfW41b5onhB65Mz 9fhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8A4UEJDhUSBySq3W4OssQpyAn/HvTVRLW0A+ROqVazQ=; b=XbBWJH4+59J+1+clfX8g+usOIjCR7OTava2sYjo9mfTRVlZjO7LApuj3GbD4A73v8v EAfLb5EqKL3WsS62BLOGkb74RIS6ZCviFug8wooO/gUsFZmsbNS6MxUxcaGoMVZyWmZX pFLVkGgpMuKx6wNZZCyoJCaZT2TLkDCOaT49v9AQ6p6X2teqCfv9iYttd+RIo7vLsiW0 aiuQwBbGnA0EVnzPeHwMludtmla2ZlYgeuv9Pk/1zl/3nH3IK+kLwuDIFxiPONj+t/md rRgwuQ/f/xN9PdEgKVfEu1mkXXcf7E4ny0kOPxPTjIwSQMKWOEhM1pw33PI2Peo+IZui L1DA== X-Gm-Message-State: AOAM5312neKt6SXYEMRbVoRDtwCgeTrngnmKOprV/of8VDdCZEqzhatT RkgwZXnxHR5/z40AlIUFUADvoE5cxv38pX8bYUDAKQgwrS8= X-Google-Smtp-Source: ABdhPJxM9X6XdbKAm+rnsbumtNDBPynSUHymR/kJNEo+1nlZYkAG3/SwkXgV8fR+m377Ytkoo53OmxLpGKDs2dyhmpo= X-Received: by 2002:a05:6122:78c:: with SMTP id k12mr27347502vkr.22.1636746784041; Fri, 12 Nov 2021 11:53:04 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <9FE99EEF-37C5-43D1-AC9D-17F3EDA19606@distal.com> <09989390-FED9-45A6-A866-4605D3766DFE@distal.com> <4E5511DF-B163-4928-9CC3-22755683999E@distal.com> <42006135.15.1636709757975@mailrelay> <7B41B7D7-0C74-4F87-A49C-A666DB970CC3@distal.com> <4008C512-31F1-4BE3-B674-A270CF674757@distal.com> In-Reply-To: <4008C512-31F1-4BE3-B674-A270CF674757@distal.com> From: Warner Losh Date: Fri, 12 Nov 2021 12:52:52 -0700 Message-ID: Subject: Re: swap_pager: cannot allocate bio To: Chris Ross Cc: Ronald Klop , freebsd-fs Content-Type: multipart/alternative; boundary="000000000000a67ddf05d09ccebc" X-Rspamd-Queue-Id: 4HrTlh4bg0z3qMZ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; TAGGED_RCPT(0.00)[freebsd]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: Y --000000000000a67ddf05d09ccebc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Nov 12, 2021 at 12:50 PM Chris Ross wrote: > > > > On Nov 12, 2021, at 11:15, Warner Losh wrote: > > So the root cause of this problem is well known. You have a memory > shortage, so you want to page out dirty pages to reclaim memory. > > However, there's not enough memory to allocate the structures you need > to do I/O and so the swapout I/O fails half way down > > the stack not being able to allocate a bio. Some paths through the > swapper cope with this well, other parts that execute less > > often cope less well. > > > > There's some hacks in the tree today to help with the GELI case: we > prioritize swapping I/O. But there's no g_alloc_bio_swapping() interface > > for swapping I/O to get priority on allocating a bio to start with. > Places that use g_clone_bio() could have the clone's copy allocated > > from a special swap pool, but that starts to get messy and isn't done > today. And the upper layers like geom_cfs and ZFS are > > inconsistent in allocations, so there's work needed to make it robust i= n > ZFS, but I have only a vague notion of what's needed. At the very > > least, the swapping I/O that comes into the top of ZFS won't have > swapping I/O marked coming out the bottom because the > > BIO_SWAP flag is quite new. > > > > So until then, swapping on zvols is fraught with deadlocks like this an= d > in the past there's been a strong admonishment > > against it. > > Apologies, Warner, but I=E2=80=99m not sure I=E2=80=99m understanding thi= s last > statement. If you mean swapping _onto_ zvols, I=E2=80=99m not doing that= . If you > mean swapping in any way, while having zvols, then yes I am doing that. > > My swap is on a partition on the non-ZFS disk. A physical disk as far as > the kernel knows, hardware RAID1. > > # pstat -s > Device 1K-blocks Used Avail Capacity > /dev/da0p3 445682648 1018524 444664124 0% > OK. That's well supported and should work w/o some of the issues that I raised. I'd misunderstood and thought you were swapping to zvols... > Let me know if what you=E2=80=99re saying above is true to my case, and a= ny advice > as to how I can avoid it. I had a =E2=80=9Cnot enough swap space=E2=80= =9D a while back, > and accordingly increased the size of my swap partition. I have 128GB of > memory, though between the ARC and the big process I was running, that > fills it easily. > Yea, this is a 'memory is exhausted' problem, and more swap won't help that. It's unclear why we run out so fast, and why the separate zones for the bio isn't providing a good level of insulation from out of memory scenarios. Warner --000000000000a67ddf05d09ccebc--