malloc does not return null when out of memory

[Gabor]

We have a little soekris box running freebsd that uses racoon for key
management.  It's used for setting up an ipsec tunnel.  I noticed that
one of these devices lost the tunnel this morning.  I looked in the
log and saw this

Jul 23 01:37:57 m0n0wall /kernel: pid 80 (racoon), uid 0, was killed: out of swap space

I reproduced this problem using this code.

#include <stdlib.h>
int
main(int argc, char **argv)
{
        char *p;
        int cnt = 0;

        while (1) {
                p = malloc(100000);
                printf("%d %p\n", ++cnt, p);
                if (p)
                        memset(p, 0, 100000);
                else
                        break;
        }
        return 0;
}

Here is the tail end of the output.  It dies when trying to poke at
the memory using memset.  If I just malloc without the memset, it
never even dies.

210 0x94b5000
211 0x94ce000
212 0x94e7000
213 0x9500000
214 0x9519000
215 0x9532000
216 0x954b000
217 0x9564000
218 0x957d000
219 0x9596000
220 0x95af000
221 0x95c8000
Killed

The kernel for this has the NO_SWAPPING option enabled since there is
essentially no disk, it boots from a CF.