Re: swap_pager: cannot allocate bio

On Thu, 11 Nov 2021 05:43:46 +0100, Chris Ross <cross+freebsd@distal.com=
>  =

wrote:

>
>
>> On Nov 10, 2021, at 23:35, Chris Ross <cross+freebsd@distal.com> wrot=
e:
>>
>> Hey all.  I have a system that I=E2=80=99m trying to do some intensiv=
e CPU and  =

>> I/O on.  FreeBSD 13.0-RELEASE, amd64, 128GB RAM, hardware RAID1 OS  =

>> volume, and a large (40TB) zpool where most of the I/O is happening.
>>
>> Initially, it was failing for me because it was running out of swap  =

>> space.  It had only the normal small (4-8G) swap partition, so I  =

>> resized the filesystems on the root disk and now have 400+GB swap.  T=
he  =

>> system had frozen up and I wasn=E2=80=99t able to log in.  When I go =
to the  =

>> console, I find a long list of:
>>
>> swap_pager: cannot allocate bio
>>
>> lines.  I was able to log into the console as root and pstat -s shows=
  =

>> the swap minimally used (7.5GB used).  Attempting a =E2=80=9Czpool st=
atus=E2=80=9D at  =

>> that point locked up.  I don=E2=80=99t know if the problem is the mem=
ory  =

>> subsystem, or zfs.
>>
>> But, based on the error, is there perhaps some kernel parameter I can=
  =

>> tune that might prevent the swap pager from encountering that error?
>
> Moving to freebsd-fs.  More information makes it looks more like a ZFS=
  =

> problem than anything else.
>
> I am able to log into another root virtual console, and I can run ps  =

> (shows many things, including dozens of "cron: running job (cron)=E2=80=
=9D jobs,  =

> in D state), and I=E2=80=99m able to wander around the root disk (3T u=
fs  =

> filesystem) without trouble.  But, as mentioned above the =E2=80=9Czpo=
ol status=E2=80=9D  =

> is hung, and I suspect if I tried to access anything in that filesyste=
m  =

> it would hang to.  Those cron jobs, which aren=E2=80=99t anything I ad=
ded, I  =

> assume are just system =E2=80=9Ccheck around the system=E2=80=9D cron =
jobs that are  =

> getting stuck there.
>
> So, if anyone has any suggestions.  I can leave this system stuck like=
  =

> this for a little while, but I=E2=80=99ll probably want to bring it ba=
ck before  =

> the end of the day tomorrow.  (I=E2=80=99m US EST, so it=E2=80=99s alm=
ost midnight  =

> here.  I=E2=80=99ll check in on email for suggestions or ideas in the =
morning.)
>
> Thanks all.
>
>                 - Chris
>


Can you press ctrl-t on the hanging process? That should print the  =

stacktrace indicating where it is waiting on.

Ronald.