kmem_map too small with ZFS and 8.2-RELEASE
Joshua Boyd
boydjd at jbip.net
Sun Mar 6 18:48:19 UTC 2011
On Sun, Mar 6, 2011 at 4:04 AM, Jeremy Chadwick
<freebsd at jdc.parodius.com> wrote:
> On Sun, Mar 06, 2011 at 02:59:43AM -0500, Joshua Boyd wrote:
>> On Fri, Mar 4, 2011 at 5:56 AM, Jeremy Chadwick
>> <freebsd at jdc.parodius.com> wrote:
>> > If you get better performance -- really, truly, honestly -- with
>> > prefetch enabled on your system, then I strongly recommend you keep it
>> > enabled. ?However, for what it's worth (probably not much), this is the
>> > first I've ever heard of a FreeBSD system performing better with
>> > prefetch enabled.
>>
>> I just recently turned it on after having it turned off for a long
>> time ... my speeds went from ~300MB/s to 600+MB/s in bonnie++. This is
>> a dual core AM3 system with 8GB of ram, and 15 disks in a striped
>> raidz configuration (3 sets striped).
>
> Here are some numbers for you. This is from a 8.2-STABLE (RELENG_8)
> system built Thu Feb 24 22:06:45 PST 2011, type amd64.
Interesting results. My kernel currently has a build date 2 days
earlier than yours. Here are my results, showing the huge increase in
speed. The only major configuration difference appears that I've
disabled the ZIL and you have yours enabled. That shouldn't make any
difference for read speeds though.
FreeBSD foghornleghorn.res.openband.net 8.2-PRERELEASE FreeBSD
8.2-PRERELEASE #13: Tue Feb 22 17:39:03 EST 2011
root at foghornleghorn.res.openband.net:/usr/obj/usr/src/sys/FOGHORNLEGHORN
amd64
/boot/loader.conf
======================
vfs.zfs.zil_disable="1"
vfs.zfs.vdev.min_pending="1"
vfs.zfs.vdev.max_pending="1"
vm.kmem_size="8192M"
vfs.zfs.arc_max=6144M
vfs.zfs.prefetch_disable="0"
vfs.zfs.txg.timeout="5"
/etc/sysctl.conf
======================
kern.maxfiles=65536
kern.maxfilesperproc=32768
vfs.read_max=32
vfs.ufs.dirhash_maxmem=16777216
kern.maxvnodes=250000
vfs.zfs.txg.write_limit_override=1073741824
ZFS details
======================
# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
tank 18.2T 7.73T 10.4T 42% ONLINE -
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 6.17T 8.10T 36.7K /tank
tank/downloads 4.89T 8.10T 2.30T /tank/downloads
tank/downloads/movies 2.59T 8.10T 2.59T /tank/downloads/movies
tank/usr 1.29T 8.10T 32.0K /tank/usr
tank/usr/home 1.29T 8.10T 69.5K /usr/home
tank/usr/home/josh 1.29T 8.10T 13.4G /usr/home/josh
tank/usr/home/josh/hellanzb 32.0K 8.10T 32.0K /usr/home/josh/hellanzb
tank/usr/home/josh/rtorrent 1.27T 8.10T 1.27T /usr/home/josh/rtorrent
tank/usr/home/josh/watch 8.00M 8.10T 8.00M /usr/home/josh/watch
# zpool status tank
pool: tank
state: ONLINE
scrub: scrub completed after 7h43m with 0 errors on Sun Mar 6 07:43:56 2011
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da8 ONLINE 0 0 0
da18 ONLINE 0 0 0
da19 ONLINE 0 0 0
da6 ONLINE 0 0 0
da7 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da11 ONLINE 0 0 0
da10 ONLINE 0 0 0
da17 ONLINE 0 0 0
da9 ONLINE 0 0 0
da5 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da3 ONLINE 0 0 0
da2 ONLINE 0 0 0
da4 ONLINE 0 0 0
errors: No known data errors
Controller details
======================
mpt0: <LSILogic SAS/SATA Adapter> port 0x6000-0x60ff mem
0xf75fc000-0xf75fffff,0xf75e0000-0xf75effff irq 18 at device 0.0 on
pci1
mpt0: [ITHREAD]
mpt0: MPI Version=1.5.20.0
mpt1: <LSILogic SAS/SATA Adapter> port 0x7000-0x70ff mem
0xf78fc000-0xf78fffff,0xf78e0000-0xf78effff irq 19 at device 0.0 on
pci2
mpt1: [ITHREAD]
mpt1: MPI Version=1.5.20.0
mpt2: <LSILogic SAS/SATA Adapter> port 0xd000-0xd0ff mem
0xf7ffc000-0xf7ffffff,0xf7fe0000-0xf7feffff irq 19 at device 0.0 on
pci6
mpt2: [ITHREAD]
mpt2: MPI Version=1.5.19.0
Disk details
======================
da8 at mpt0 bus 0 scbus0 target 0 lun 0
da8: <ATA ST31000528AS CC38> Fixed Direct Access SCSI-5 device
da8: 300.000MB/s transfers
da8: Command Queueing enabled
da8: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da9 at mpt0 bus 0 scbus0 target 1 lun 0
da9: <ATA Hitachi HDS72101 A39C> Fixed Direct Access SCSI-5 device
da9: 300.000MB/s transfers
da9: Command Queueing enabled
da9: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da0 at mpt1 bus 0 scbus1 target 0 lun 0
da0: <ATA Hitachi HDS5C302 A580> Fixed Direct Access SCSI-5 device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da10 at mpt0 bus 0 scbus0 target 2 lun 0
da10: <ATA SAMSUNG HD103SJ 00E4> Fixed Direct Access SCSI-5 device
da10: 300.000MB/s transfers
da10: Command Queueing enabled
da10: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da11 at mpt0 bus 0 scbus0 target 3 lun 0
da11: <ATA SAMSUNG HD103SJ 00E4> Fixed Direct Access SCSI-5 device
da11: 300.000MB/s transfers
da11: Command Queueing enabled
da11: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da1 at mpt1 bus 0 scbus1 target 1 lun 0
da1: <ATA Hitachi HDS5C302 A580> Fixed Direct Access SCSI-5 device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da2 at mpt1 bus 0 scbus1 target 2 lun 0
da2: <ATA Hitachi HDS5C302 A580> Fixed Direct Access SCSI-5 device
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da3 at mpt1 bus 0 scbus1 target 3 lun 0
da3: <ATA Hitachi HDS5C302 A580> Fixed Direct Access SCSI-5 device
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da4 at mpt1 bus 0 scbus1 target 4 lun 0
da4: <ATA Hitachi HDS5C302 A580> Fixed Direct Access SCSI-5 device
da4: 300.000MB/s transfers
da4: Command Queueing enabled
da4: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da5 at mpt1 bus 0 scbus1 target 5 lun 0
da5: <ATA SAMSUNG HD103SI 1118> Fixed Direct Access SCSI-5 device
da5: 300.000MB/s transfers
da5: Command Queueing enabled
da5: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da6 at mpt1 bus 0 scbus1 target 6 lun 0
da6: <ATA ST31000340AS SD04> Fixed Direct Access SCSI-5 device
da6: 300.000MB/s transfers
da6: Command Queueing enabled
da6: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da7 at mpt1 bus 0 scbus1 target 7 lun 0
da7: <ATA SAMSUNG HD103SI 1118> Fixed Direct Access SCSI-5 device
da7: 300.000MB/s transfers
da7: Command Queueing enabled
da7: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da16 at mpt2 bus 0 scbus2 target 82 lun 0
da16: <ATA ST31000340AS SD04> Fixed Direct Access SCSI-5 device
da16: 300.000MB/s transfers
da16: Command Queueing enabled
da16: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da17 at mpt2 bus 0 scbus2 target 83 lun 0
da17: <ATA Hitachi HDS72101 A39C> Fixed Direct Access SCSI-5 device
da17: 300.000MB/s transfers
da17: Command Queueing enabled
da17: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da18 at mpt2 bus 0 scbus2 target 84 lun 0
da18: <ATA SAMSUNG HD103SI 1118> Fixed Direct Access SCSI-5 device
da18: 300.000MB/s transfers
da18: Command Queueing enabled
da18: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da19 at mpt2 bus 0 scbus2 target 85 lun 0
da19: <ATA SAMSUNG HD103SI 1118> Fixed Direct Access SCSI-5 device
da19: 300.000MB/s transfers
da19: Command Queueing enabled
da19: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
Benchmark results #1
======================
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
foghornleghorn. 16G 213 99 266782 53 90296 19 480 95 218719
24 229.7 6
Latency 43348us 37929us 242ms 102ms 68306us 462ms
Version 1.96 ------Sequential Create------ --------Random Create--------
foghornleghorn.res. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 15220 44 +++++ +++ 19214 56 22371 58 +++++ +++ 22133 66
Latency 10658us 60us 82us 6540us 39us 1677us
Benchmark results #2
======================
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
foghornleghorn. 16G 201 99 276506 56 198428 38 459 97 627451
73 252.0 5
Latency 45695us 35953us 265ms 69630us 42440us 389ms
Version 1.96 ------Sequential Create------ --------Random Create--------
foghornleghorn.res. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 14988 50 +++++ +++ 18693 59 18535 51 +++++ +++ 20827 67
Latency 13309us 93us 116us 8165us 36us 1046us
--
Joshua Boyd
JBipNet
E-mail: boydjd at jbip.net
http://www.jbip.net
More information about the freebsd-fs
mailing list