kmem_map too small with ZFS and 8.2-RELEASE

[Jeremy Chadwick]

On Sun, Mar 06, 2011 at 02:59:43AM -0500, Joshua Boyd wrote:
> On Fri, Mar 4, 2011 at 5:56 AM, Jeremy Chadwick
> <freebsd at jdc.parodius.com> wrote:
> > If you get better performance -- really, truly, honestly -- with
> > prefetch enabled on your system, then I strongly recommend you keep it
> > enabled. ?However, for what it's worth (probably not much), this is the
> > first I've ever heard of a FreeBSD system performing better with
> > prefetch enabled.
> 
> I just recently turned it on after having it turned off for a long
> time ... my speeds went from ~300MB/s to 600+MB/s in bonnie++. This is
> a dual core AM3 system with 8GB of ram, and 15 disks in a striped
> raidz configuration (3 sets striped).

Here are some numbers for you.  This is from a 8.2-STABLE (RELENG_8)
system built Thu Feb 24 22:06:45 PST 2011, type amd64.


/boot/loader.conf
======================
	ahci_load="yes"
	vm.kmem_size="8192M"
	vfs.zfs.arc_max="6144M"
	vfs.zfs.prefetch_disable="1"
	vfs.zfs.txg.timeout="5"

/etc/sysctl.conf
======================
	kern.maxvnodes=250000
	vfs.zfs.txg.write_limit_override=1073741824

ZFS details
======================
# zpool list
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
data  2.72T   780G  1.96T    28%  ONLINE  -

# zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
data           519G  1.28T  28.0K  none
data/backups   519G  1.28T   519G  /backups
data/home     7.19M  1.28T  7.19M  /home

# zpool status data
  pool: data
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ada1    ONLINE       0     0     0
            ada2    ONLINE       0     0     0
            ada3    ONLINE       0     0     0

Pool and filesystem attributes are all defaults (ZFS v15),
except for mountpoint.

Disk details
======================
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD1001FALS-00U9B0 05.00K05> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD1001FALS-00U9B0 05.00K05> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)

Benchmark commands
======================
	shutdown -r now
	<wait for machine to come back up>
	cd /backups && bonnie++ -s 16g -u root

Benchmark results #1
======================
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
XXXXX           16G    92  99 127026  42 62634  18   244  99 177522  22 207.0   5
Latency               144ms    1277ms    1616ms   47845us     111ms     639ms
Version  1.96       ------Sequential Create------ --------Random Create--------
XXXXX               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18945  96 +++++ +++ 17552  98 18975  94 +++++ +++ 18286  98
Latency             18018us      85us     131us   17862us      38us      80us

Benchmark results #2
======================
Results with only one change made to /boot/loader.conf:

	vfs.zfs.prefetch_disable="0"

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
XXXXX           16G    95  99 126656  40 74678  24   229  97 187918  26 219.8   5
Latency               159ms    1813ms    2506ms     330ms     308ms     657ms
Version  1.96       ------Sequential Create------ --------Random Create--------
XXXXX               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 20140  96 +++++ +++ 16814  96 10431  54 +++++ +++ 14846  95
Latency             21010us     158us    1242us     146ms     721us   11662us

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |