BKVASIZE for large block-size filesystems

[Sven Willenberger]

On Thu, 2005-05-26 at 09:02 -0400, Sven Willenberger wrote:
> On Thu, 2005-05-26 at 10:38 +1000, Bruce Evans wrote:
> > On Wed, 25 May 2005, Sven Willenberger wrote:
> > 
<<SNIP>>
> > 
> > > My question is are there any caveats about increasing BKVASIZE to 65536?
> > > The system has 8G of RAM and I understand that nbufs decreases with
> > > increasing BKVASIZE;
> > 
> > The decrease in nbufs is a bug.  It defeats half of the point of increasing
> > BKVASIZE: if most buffers have size 64K, then increasing BKVASIZE from 16K
> > to 64K gives approximately nbuf/4 buffers all of size 64K instead of nbuf
> > buffers, with nbuf/4 of them of size 64K and 3*nbuf/4 of them unusable.
> > Thus it avoids some resource wastage at a cost of possibly not using enough
> > resources for effective caching.  However, little is lost if most buffers
> > have size 64K.  Then the reduced nbuf consumes all of the kva resources that
> > we are willing to allocate.  The problem is when file systems are mixed and
> > ones with a block size of 64K are not used much or at all.  The worst case
> > is when all blocks have size 512, which can happen for msdosfs.  Then up
> > to (BKVASIZE - 512) / BKVASIZE of the kva resource is wasted (> 99% for
> > BKVASIZE = 65536 but only 97% for BKVASIZE = 16384).
> > 
> > To fix the bug, change BKVASIZE in kern_vfs_bio_buffer_alloc() to 16384
> > and consider adjusting the machbcache tunable (see below).
> > 
> 

Alas this did not work ... I replaced BKVASIZE in that function (and
only that function ) with 16384 and adjusted BKVASIZE to 65536 in
sys/param.h. After make buildworld, make buildkernel, make installkernel
I rebooted (attempted to into single user mode). However, after the post
and the BSD splash screen (where I am prompted to choose single user)
the machine gets almost nowhere into the boot process and gets as far
as:

Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights
reserved.
FreeBSD 5.4-STABLE #1: Sat May 14 17:37:43 EDT 2005
    root at myserver:/usr/obj/usr/src/sys/NORACCT
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Opteron(tm) Processor 248 (2190.75-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0xf5a  Stepping = 10

Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  AMD Features=0xe0500800<SYSCALL,NX,MMX+,LM,3DNow+,3DNow>
real memory  = 8589934592 (8192 MB)
avail memory = 8255700992 (7873 MB)


then the console completely locks up and there is no further response.
Booting kernel.old does work fine and it proceeds to the next section
(ACPI APIC table).

> Ahh, so this is literal replace the word "BKVASIZE" in that function
> with the word "16384". I am assuming that I can leave other instances of
> BKVASIZE and BKVAMASK in that file (vfs_bio.c) alone then?
> 
> > > how can I either determine if the resulting nbufs
> > > will be sufficient or calculate what is needed based on RAM and system
> > > usage?
> > 
> > nbuf is not directly visible except using a debugger, but vfs.maxbufspace
> > gives it indirectly -- divide the latter by BKVASIZE to get nbuf.  A few
> > thousand for it is plenty.
> > 
> > I used to use BKVASIZE = 65536, and fixed the bug as above, and also doubled
> > nbuf in kern_vfs_bio_buffer_alloc(), and also configured VM_BCACHE_SIZE_MAX
> > to 512M so that the elevated nbuf was actually used, but the need for
> > significantly increasing the default nbuf (at least with BKVASIZE = 16384)
> > went away many years ago when memory sizes started exceeding 256M or so.
> > My doubling of nbuf broke a few years later when memory sizes started
> > exceeding 1GB.  i386's just don't have enough virtual address space to use
> > a really large nbuf, so when there is enough physical memory the default
> > nbuf is as large as possible.  I was only tuning BKVASIZE and
> > VM_BCACHE_SIZE_MAX to benchmark file systems with large block sizes, but
> > the performance with large block sizes was poor even with this tuning so
> > I lost interest in it.  Now I just use the defaults and the bug fix
> > reduces to a spelling change.  nbuf defeaults to about 7000 on my machines
> > with 1GB of memory.  This is plenty.  With BKVASIZE = 64K and without the
> > fix, it would be 1/4 as much, which seems a little low.
> > 

I may have to try some of these other tricks and see what I can do ..
first I need to build a regular kernel again so I can have a "safe"
kernel.old after trying some further changes.

Sven