numa involved in instability and swap usage despite RAM free?

[Alexander Leidinger]

Hi,

I don't have hard evidence, but there is enough "smell" to open up a  
discussion...

Short:
Can it be that enabling numa in the kernel is the reason why some  
people see instability with zfs and usage of swap while a lot of free  
RAM is available?

Long:
I have a dual-socket Xeon system (E5620 + L5630... yes, not the same,  
but compatible enough to be able to run together) with 64 GB RAM. I  
run -current on it (currently it's at r333966 and it was for all the  
tests below).

What I see with numa enabled and no zfs patches is, that at some point  
I have half the RAM free, swap is in use, and after a lot of compiling  
ports in different jails ZFS comes to a halt (sometimes I can unblock  
by killing a compile, sometimes I can't even kill, only way out is  
power-cycle). I've seen this around twice a week.

When I keep numa enabled and have applied this ZFS patch  
https://reviews.freebsd.org/D7538 the bahavior changes. AFter a while  
half of the RAM is free, swap is in use, and after enough compiling  
ports in jails I get a panic (unfortunately not enough debug info in  
the textdump to know exactly what he problem is).

Since 2 weeks I have numa compiled out of the kernel (and still the  
ZFS patch inside). The system is down to 17 GB free and NO swap in  
use. I'm compiling ports in 16 jails (one of them with parts of KDE5 =  
currently about 700 ports compiled) and not a single issue like the  
above.

For everyone with swap issues or ZFS issues similar to the ones I  
see... do you have numa enabled and can you please try without and  
report back?

Can it be that if memory request can not be fulfilled from one numa  
domain, there is no fallback to another numa domain for all the  
various kinds of memory allocation we have in the kernel  
(contigmem/no-sleep/...)?

Bye,
Alexander.

-- 
http://www.Leidinger.net Alexander at Leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.org    netchild at FreeBSD.org  : PGP 0x8F31830F9F2772BF
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: Digitale PGP-Signatur
URL: <http://lists.freebsd.org/pipermail/freebsd-current/attachments/20180624/81d9e189/attachment.sig>