Observations on virtual memory operations

[doug]

I have two servers running jails that "routinely" run out of swapspace with
no demand paging activity. To try and get a handle on VM/swapspace
management I have been tracking swapinfo vs memory use as measured by top.
The numbers do not exactly add up but I assume that is not involved in my
issue.

The system:
   real memory  = 8589934592 (8192 MB)
   avail memory = 8248696832 (7866 MB)

In top the sum of Active, Inactive, Laundry, Wired, Buf and Free is almost
constant between 8704 and 8706 MB. Wired+Buf is fairly constant around
2200MB. Free has no relation to anything as long as the system is working
(e.g., what you would expect).

The other day I caught the system at 73% swapspace used. At this level the
system was in a near thrashing state in that typing a key got it echoed in
10 <--> 30 seconds. There was about 600MB of swapspace at this point. I
would think there is no way to debug this except as a thought experiment.

My other thought is it would be nice if after perhaps 5,000 -> 10,000 'out
of swap space message' logging could stop. I read a google post suggesting
this can be handled by logging the console output to a file. Does anyone
have experience with this?

Right now we are surviving by email warnings when the system gets to 50%
swap used and monitoring,