vmdaemon CPU usage and poor performance in 10.0-RELEASE

[Polyack, Steve]

All,

We have a handful of database servers running FreeBSD 10.0-RELEASE and PostgreSQL 9.3.4.  The servers have 128GB or 256GB of RAM.  After a short period of running and some "light" load (load average of 6 on a 16-core w/ HT box), the system performance becomes terrible.  Logging in and simply attempting to run some commands results in the command prompt hanging for several seconds before anything is returned (even for something simple like `test`).  If we run `top -SH` when the system performance is bad, we can see the vmdaemon thread/kernel process is using 100% CPU.  Eventually the vmdaemon CPU usage drops and system performance seems to return to normal, but this doesn't last long.

Has anyone seen this behavior?  I found a bug that seems to describe the same problem (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=190300), but it's been open for 2 months without any responses.  For what it's worth, the problem does not seem as bad when we reduce postgresql shared buffers from 128GB to 8GB, but it's still there.

I can't imagine what vmdaemon is spending all of its time doing - `top -SH` output and memory usage looks like:

last pid: 97787;  load averages:  6.03,  6.17,  6.03                                                                                             up 19+15:46:19  08:50:07
345 processes: 41 running, 220 sleeping, 78 waiting, 6 lock
CPU: 16.0% user,  0.0% nice, 10.0% system,  0.0% interrupt, 74.0% idle
Mem: 6314M Active, 234G Inact, 4078M Wired, 2720M Cache, 1704M Buf, 2719M Free
Swap: 16G Total, 329M Used, 16G Free, 2% Inuse

  PID USERNAME      PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   11 root          155 ki31     0K   512K CPU26  26 448.8H  95.07% idle{idle: cpu26}
   11 root          155 ki31     0K   512K CPU29  29 449.1H  92.48% idle{idle: cpu29}
    6 root          -16    -     0K    16K CPU27  27  81.5H  89.70% vmdaemon
   11 root          155 ki31     0K   512K RUN     1 459.2H  89.60% idle{idle: cpu1}
   11 root          155 ki31     0K   512K RUN    14 452.9H  89.26% idle{idle: cpu14}
   11 root          155 ki31     0K   512K CPU5    5 458.7H  89.16% idle{idle: cpu5}
   11 root          155 ki31     0K   512K CPU31  31 449.1H  88.96% idle{idle: cpu31}
   11 root          155 ki31     0K   512K RUN     6 458.2H  88.87% idle{idle: cpu6}
   11 root          155 ki31     0K   512K CPU11  11 453.0H  88.38% idle{idle: cpu11}
   11 root          155 ki31     0K   512K RUN     3 457.9H  88.28% idle{idle: cpu3}
   11 root          155 ki31     0K   512K CPU12  12 452.8H  87.89% idle{idle: cpu12}
   11 root          155 ki31     0K   512K CPU2    2 458.6H  87.26% idle{idle: cpu2}
   11 root          155 ki31     0K   512K CPU7    7 458.6H  87.16% idle{idle: cpu7}
   11 root          155 ki31     0K   512K CPU10  10 458.5H  86.87% idle{idle: cpu10}
   11 root          155 ki31     0K   512K CPU15  15 453.8H  86.47% idle{idle: cpu15}
   11 root          155 ki31     0K   512K CPU13  13 453.6H  85.79% idle{idle: cpu13}
   11 root          155 ki31     0K   512K CPU4    4 458.4H  85.69% idle{idle: cpu4}
   11 root          155 ki31     0K   512K RUN     9 453.4H  85.35% idle{idle: cpu9}
   11 root          155 ki31     0K   512K CPU0    0 460.5H  85.25% idle{idle: cpu0}
   11 root          155 ki31     0K   512K CPU17  17 449.3H  84.08% idle{idle: cpu17}
   11 root          155 ki31     0K   512K RUN    25 449.0H  83.40% idle{idle: cpu25}
   11 root          155 ki31     0K   512K CPU18  18 448.9H  82.76% idle{idle: cpu18}
   11 root          155 ki31     0K   512K CPU16  16 448.8H  82.18% idle{idle: cpu16}
   11 root          155 ki31     0K   512K CPU23  23 448.8H  81.88% idle{idle: cpu23}
   11 root          155 ki31     0K   512K CPU8    8 454.3H  81.30% idle{idle: cpu8}
   11 root          155 ki31     0K   512K RUN    21 449.1H  81.05% idle{idle: cpu21}
   11 root          155 ki31     0K   512K CPU20  20 448.6H  80.96% idle{idle: cpu20}
   11 root          155 ki31     0K   512K RUN    22 448.7H  80.66% idle{idle: cpu22}
   11 root          155 ki31     0K   512K RUN    30 448.4H  79.59% idle{idle: cpu30}
   11 root          155 ki31     0K   512K CPU24  24 448.7H  79.39% idle{idle: cpu24}
   11 root          155 ki31     0K   512K RUN    19 449.2H  78.47% idle{idle: cpu19}
   11 root          155 ki31     0K   512K CPU28  28 448.7H  54.20% idle{idle: cpu28}

$ uname -a
FreeBSD db00 10.0-RELEASE-p7 FreeBSD 10.0-RELEASE-p7 #2: Thu Jul 24 14:35:07 EDT 2014     root at build10:/usr/obj/usr/src/sys/GENERIC  amd64

Any help would be appreciated.  We would be more than happy to answer any other questions.

Thanks,

Steve Polyack