madvise(MADV_FREE) doesn't work in some cases?

From: Vitaliy Gusev <gusev.vitaliy_at_gmail.com>
Date: Fri, 02 Jul 2021 23:56:40 UTC
Hi, 

I came across not expected behaviour with madvise() in FreeBSD.

Attached test program mmapfork does: mmap, fork, touch memory and then madvise(MADV_FREE).

Expected behaviour - one process can allocate memory (lazy allocation) while system is freeing previously allocated memory for a second process.

Current behaviour -  system kills one process with message in dmesg:

pid 31314 (mmapfork), jid 0, uid 1001, was killed: out of swap space

Running this test in Linux or illumos shows expected behaviour with a little difference in illumos - it frees memory almost immediately, w/o needs lack of memory in a system.

If use MADV_NOTNEED - no changes.

If modify program and do not do fork(), but run two instances  - that shows expected behaviour.

To reproduce just disable swap, and run program with argument as 1/2 RAM on a system. For instance, command below will try run and use ~ 2GB area twice.

[vetal@bsdev ~]$ ./mmapfork 2000

Testing program is attached.

Note, during testing I disabled swap on all systems: Linux, illumos and FreeBSD.

Does it mean madvise() doesn't work well in FreeBSD or test does something wrong?

Thanks,
Vitaliy Gusev