Re: swap_pager: cannot allocate bio

On 11/12/21 11:59, Chris Ross wrote:
>
>> On Nov 12, 2021, at 14:52, Warner Losh <imp@bsdimp.com> wrote:
>> My swap is on a partition on the non-ZFS disk.  A physical disk as far as the kernel knows, hardware RAID1.
>>
>> # pstat -s
>> Device           1K-blocks   Used    Avail  Capacity
>> /dev/da0p3      445682648 1018524 444664124 0%
>>
>> OK. That's well supported and should work w/o some of the issues that I raised. I'd misunderstood and thought you were swapping to zvols...
>>   
>> Let me know if what you’re saying above is true to my case, and any advice as to how I can avoid it.  I had a “not enough swap space” a while back, and accordingly increased the size of my swap partition.  I have 128GB of memory, though between the ARC and the big process I was running, that fills it easily.
>>
>> Yea, this is a 'memory is exhausted' problem, and more swap won't help that. It's unclear why we run out so fast, and why the separate zones for the bio isn't providing a good level of insulation from out of memory scenarios.
> Okay.  Well, I can’t easily add more memory to this machine, though I am investigating it.  I certainly can’t do it in short order.  I presume the problem is that I recently increased the size of this pool by adding a large raidz vdev to it.  I’ve only been seeing this since.  Is there any way I can “limit” the perceived size of the ZFS filesystem to ease the problem?  Is there anything I can tune to help?  Can I turn off or drastically reduce the ARC?  A decrease in performance would be better than getting stuck after a day or so.  :-)

I don't think this is "the right way to do things" *but* I have begun 
using this sysctl to limit the size of my arc*.  the reason i say it's 
not the right way is because it may just paper over a real bug and 
preventing us from getting it fixed.  might be worth testing though to 
see if it helps:


# 25GB arc
vfs.zfs.arc.max=25000000000


cheers,
-pete

*my use-case is for a system running a bunch of VM's and this has 
allowed me to avoid swapping.  perf has been acceptable.

-- 
Pete Wright
pete@nomadlogic.org
@nomadlogicLA