From nobody Thu Nov 11 04:43:46 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7A839184AAFC; Thu, 11 Nov 2021 04:43:55 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from relay.wiredblade.com (relay.wiredblade.com [168.235.95.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4HqTd709LXz4tvj; Thu, 11 Nov 2021 04:43:54 +0000 (UTC) (envelope-from cross+freebsd@distal.com) Received: from mail.distal.com (pool-108-48-165-176.washdc.fios.verizon.net [108.48.165.176]) by relay.wiredblade.com with ESMTPSA (version=TLSv1.2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256) ; Thu, 11 Nov 2021 04:43:53 +0000 Received: from smtpclient.apple ( [2001:420:c0c8:1002::200]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id dbeee53b (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Wed, 10 Nov 2021 23:43:52 -0500 (EST) Content-Type: text/plain; charset=utf-8 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: swap_pager: cannot allocate bio From: Chris Ross In-Reply-To: <9FE99EEF-37C5-43D1-AC9D-17F3EDA19606@distal.com> Date: Wed, 10 Nov 2021 23:43:46 -0500 Cc: freebsd-stable@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <09989390-FED9-45A6-A866-4605D3766DFE@distal.com> References: <9FE99EEF-37C5-43D1-AC9D-17F3EDA19606@distal.com> To: freebsd-fs X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Rspamd-Queue-Id: 4HqTd709LXz4tvj X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd] X-ThisMailContainsUnwantedMimeParts: N > On Nov 10, 2021, at 23:35, Chris Ross = wrote: >=20 > Hey all. I have a system that I=E2=80=99m trying to do some intensive = CPU and I/O on. FreeBSD 13.0-RELEASE, amd64, 128GB RAM, hardware RAID1 = OS volume, and a large (40TB) zpool where most of the I/O is happening. >=20 > Initially, it was failing for me because it was running out of swap = space. It had only the normal small (4-8G) swap partition, so I resized = the filesystems on the root disk and now have 400+GB swap. The system = had frozen up and I wasn=E2=80=99t able to log in. When I go to the = console, I find a long list of: >=20 > swap_pager: cannot allocate bio >=20 > lines. I was able to log into the console as root and pstat -s shows = the swap minimally used (7.5GB used). Attempting a =E2=80=9Czpool = status=E2=80=9D at that point locked up. I don=E2=80=99t know if the = problem is the memory subsystem, or zfs. >=20 > But, based on the error, is there perhaps some kernel parameter I can = tune that might prevent the swap pager from encountering that error? Moving to freebsd-fs. More information makes it looks more like a ZFS = problem than anything else. I am able to log into another root virtual console, and I can run ps = (shows many things, including dozens of "cron: running job (cron)=E2=80=9D= jobs, in D state), and I=E2=80=99m able to wander around the root disk = (3T ufs filesystem) without trouble. But, as mentioned above the = =E2=80=9Czpool status=E2=80=9D is hung, and I suspect if I tried to = access anything in that filesystem it would hang to. Those cron jobs, = which aren=E2=80=99t anything I added, I assume are just system =E2=80=9Cc= heck around the system=E2=80=9D cron jobs that are getting stuck there. So, if anyone has any suggestions. I can leave this system stuck like = this for a little while, but I=E2=80=99ll probably want to bring it back = before the end of the day tomorrow. (I=E2=80=99m US EST, so it=E2=80=99s = almost midnight here. I=E2=80=99ll check in on email for suggestions or = ideas in the morning.) Thanks all. - Chris=