BKVASIZE for large block-size filesystems
sven at dmv.com
Fri May 27 14:48:38 PDT 2005
On Thu, 2005-05-26 at 09:02 -0400, Sven Willenberger wrote:
> On Thu, 2005-05-26 at 10:38 +1000, Bruce Evans wrote:
> > On Wed, 25 May 2005, Sven Willenberger wrote:
> > > My question is are there any caveats about increasing BKVASIZE to 65536?
> > > The system has 8G of RAM and I understand that nbufs decreases with
> > > increasing BKVASIZE;
> > The decrease in nbufs is a bug. It defeats half of the point of increasing
> > BKVASIZE: if most buffers have size 64K, then increasing BKVASIZE from 16K
> > to 64K gives approximately nbuf/4 buffers all of size 64K instead of nbuf
> > buffers, with nbuf/4 of them of size 64K and 3*nbuf/4 of them unusable.
> > Thus it avoids some resource wastage at a cost of possibly not using enough
> > resources for effective caching. However, little is lost if most buffers
> > have size 64K. Then the reduced nbuf consumes all of the kva resources that
> > we are willing to allocate. The problem is when file systems are mixed and
> > ones with a block size of 64K are not used much or at all. The worst case
> > is when all blocks have size 512, which can happen for msdosfs. Then up
> > to (BKVASIZE - 512) / BKVASIZE of the kva resource is wasted (> 99% for
> > BKVASIZE = 65536 but only 97% for BKVASIZE = 16384).
> > To fix the bug, change BKVASIZE in kern_vfs_bio_buffer_alloc() to 16384
> > and consider adjusting the machbcache tunable (see below).
Alas this did not work ... I replaced BKVASIZE in that function (and
only that function ) with 16384 and adjusted BKVASIZE to 65536 in
sys/param.h. After make buildworld, make buildkernel, make installkernel
I rebooted (attempted to into single user mode). However, after the post
and the BSD splash screen (where I am prompted to choose single user)
the machine gets almost nowhere into the boot process and gets as far
Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights
FreeBSD 5.4-STABLE #1: Sat May 14 17:37:43 EDT 2005
root at myserver:/usr/obj/usr/src/sys/NORACCT
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Opteron(tm) Processor 248 (2190.75-MHz K8-class CPU)
Origin = "AuthenticAMD" Id = 0xf5a Stepping = 10
real memory = 8589934592 (8192 MB)
avail memory = 8255700992 (7873 MB)
then the console completely locks up and there is no further response.
Booting kernel.old does work fine and it proceeds to the next section
(ACPI APIC table).
> Ahh, so this is literal replace the word "BKVASIZE" in that function
> with the word "16384". I am assuming that I can leave other instances of
> BKVASIZE and BKVAMASK in that file (vfs_bio.c) alone then?
> > > how can I either determine if the resulting nbufs
> > > will be sufficient or calculate what is needed based on RAM and system
> > > usage?
> > nbuf is not directly visible except using a debugger, but vfs.maxbufspace
> > gives it indirectly -- divide the latter by BKVASIZE to get nbuf. A few
> > thousand for it is plenty.
> > I used to use BKVASIZE = 65536, and fixed the bug as above, and also doubled
> > nbuf in kern_vfs_bio_buffer_alloc(), and also configured VM_BCACHE_SIZE_MAX
> > to 512M so that the elevated nbuf was actually used, but the need for
> > significantly increasing the default nbuf (at least with BKVASIZE = 16384)
> > went away many years ago when memory sizes started exceeding 256M or so.
> > My doubling of nbuf broke a few years later when memory sizes started
> > exceeding 1GB. i386's just don't have enough virtual address space to use
> > a really large nbuf, so when there is enough physical memory the default
> > nbuf is as large as possible. I was only tuning BKVASIZE and
> > VM_BCACHE_SIZE_MAX to benchmark file systems with large block sizes, but
> > the performance with large block sizes was poor even with this tuning so
> > I lost interest in it. Now I just use the defaults and the bug fix
> > reduces to a spelling change. nbuf defeaults to about 7000 on my machines
> > with 1GB of memory. This is plenty. With BKVASIZE = 64K and without the
> > fix, it would be 1/4 as much, which seems a little low.
I may have to try some of these other tricks and see what I can do ..
first I need to build a regular kernel again so I can have a "safe"
kernel.old after trying some further changes.
More information about the freebsd-amd64