FreeBSD 8.1-Prerelease Panic amd64 w/ZFS..

[Howard Leadmon]

> -----Original Message-----
> From: Jeremy Chadwick [mailto:freebsd at jdc.parodius.com]
> Sent: Friday, May 28, 2010 9:46 AM
> To: Andriy Gapon
> Cc: Howard Leadmon; amd64 at freebsd.org; freebsd-fs at freebsd.org
> Subject: Re: FreeBSD 8.1-Prerelease Panic amd64 w/ZFS..
> 
> On Fri, May 28, 2010 at 12:13:51PM +0300, Andriy Gapon wrote:
> > on 28/05/2010 10:30 Howard Leadmon said the following:
> > >  I know there used to be some issues with this a while back, but
> thought
> > > with the 8.x FBSD servers most of this tuned itself, or then again
> maybe
> > > this is something different.
> > >
> > >
> > >
> > >  For the first time I ever recall, I found my FreeBSD 8 server was
> crashed
> > > this past morning, with the following error:
> > >
> > >
> > >
> > > panic:kmem_malloc(131072):kmem_map to small: 1296826368 total
> allocated
> > >
> > > cupid=4
> >
> > O wow, this is an amd64 system (with 4G RAM) and you've got "kmem_map
> too
> > small".  That's very very strange.
> 
> It is?  On amd64, vm.kmem_size (not vm.kmem_size_max) is what has to be
> increased.  I can point folks to the "official" statement from pjd@ and
> some others if need be.  For a very long time I questioned this because
> for an even longer amount of time we were being told to increase
> vm.kmem_size_max.
> 
> vm.kmem_size_max, by default, is already huge on amd64 (~320GB or
> something like that).  Proof:
> 
> vm.kmem_size_max: 329853485875
> 
> To the OP: you will need to increase vm.kmem_size in /boot/loader.conf
> and reboot the system.  "What value do I pick?"  With 4GB, I would
> recommend you use these two variables:
> 
> vm.kmem_size="2048M"
> vfs.zfs.arc_max="1536M"
> 
> This will increase the available kmem, and also limit the ARC size
> explicitly to nothing larger than 1.5GB.  This should stabilise your
> system.
> 
> --
> | Jeremy Chadwick                                   jdc at parodius.com |
> | Parodius Networking                       http://www.parodius.com/ |
> | UNIX Systems Administrator                  Mountain View, CA, USA |
> | Making life hard for others since 1977.              PGP: 4BD6C0CB |


 Thanks Jeremy, I will try your recommended settings provided above.

 To the other poster, as to the settings of kmem, I had nothing specific
set, just whatever FBSD was using by default.   In loader.conf all I had
was:

zfs_load="YES"
vfs.root.mountfrom="zfs:tank/root"


As to the setting of kmem and arc, I had the following which I will assume
were defaults or auto-tunes:

vfs.zfs.arc_max : 862653440
vm.kmem_size    : 1380245504
vm.kmem_size_max: 329853485875


Like Jeremy said, it sure looks like vm.kmem_size_max is quite large by
default.

After applying Jeremy's recommendations, I now see:

vfs.zfs.arc_max : 1610612736
vm.kmem_size    : 2147483648
vm.kmem_size_max: 329853485875


I guess while we are all on the subject, I notice in the dmesg log the
message:

ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is
present;
            to enable, add "vfs.zfs.prefetch_disable=0" to
/boot/loader.conf.

Is this anything I want to enable, that's like a big performance win, or do
I just not have enough RAM to support it?   Always been kinda curious about
it, but so far I am liking ZFS, well outside of the machine panic..  LOL


---
Howard Leadmon