freebsd 9.0-release + zfs + mysqld(percona) = kernel: swap zone exhausted, increase kern.maxswzone

[Philip M. Gollucci]

/var/log/messages
Mar 23 22:21:50 sabertooth kernel: swap zone exhausted, increase
kern.maxswzone
Mar 23 22:21:50 sabertooth kernel: pid 86697 (mysqld), uid 88, was
killed: out of swap space

how to repeat:
$ mysql -ux < file.sql (~150GB) worth

basically, it slows down continually until it dies.  IF you (suspend)
the process in time it recovers some, but eventually you have to suspend
it every 1s for ~3 minutes.  The load is ~10 at this point.

I've looked at top, ps, iostat, zpool iostat, vmstat -z, vmstat -m
and I don't see anything wonky.  I can provide more info on request.

system description:

$ df
zmysqlD    801G    658G    142G    82%    /var/db/mysql/data
zmysqlL    133G     26G    107G    20%    /var/db/mysql/log

its a 600GB innodb space, mysql has
innodb_buffer_pool_size = 80GB
about 1GB of data is MyISAM the rest is InnoDB

The machine has 96GB of RAM

$ cat /etc/fstab
/dev/gpt/swap0  none    swap    sw      0       0
/dev/gpt/swap1  none    swap    sw      0       0

tmpfs   /tmp            tmpfs   rw      2       0

swapinfo -h will show %6 and %6 usage on the swap devices
/tmp remains < 5% used

$ grep maxswzone /boot/loader.conf
kern.maxswzone="67108864"  ## double the default

$ gpart show
=>       34  286749421  da3  GPT  (136G)
         34        128    1  freebsd-boot  (64k)
        162  201326592    2  freebsd-swap  (96G)
  201326754   85422701    3  freebsd-zfs  (40G)

=>       34  286749421  da4  GPT  (136G)
         34        128    1  freebsd-boot  (64k)
        162  201326592    2  freebsd-swap  (96G)
  201326754   85422701    3  freebsd-zfs  (40G)

da[012] are SSDs, the rest are 15krpm

$ zpool status
  pool: zmysqlD
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zmysqlD     ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0
            da12    ONLINE       0     0     0
            da13    ONLINE       0     0     0
            da14    ONLINE       0     0     0
        logs
          da0       ONLINE       0     0     0
        cache
          da2       ONLINE       0     0     0

errors: No known data errors

  pool: zmysqlL
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zmysqlL     ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
        cache
          da1       ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da3p3   ONLINE       0     0     0
            da4p3   ONLINE       0     0     0



-- 
------------------------------------------------------------------------
1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70  3F8C 75B8 8FFB DB9B 8C1C
Philip M. Gollucci (pgollucci at p6m7g8.com) c: 703.336.9354
Member,                           Apache Software Foundation
Committer,                        FreeBSD Foundation
Consultant,                       P6M7G8 Inc.
Director Operations,              Ridecharge Inc.

Work like you don't need the money,
love like you'll never get hurt,
and dance like nobody's watching.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
Url : http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20120326/2ec889e3/signature.pgp