Help ZFS FreeBSD 8.0 RC2 Write performance issue

[Sam Fourman Jr.]

On Wed, Nov 11, 2009 at 5:26 PM, Dan Nelson <dnelson at allantgroup.com> wrote:
> In the last episode (Nov 11), Sam Fourman Jr. said:
>> On Wed, Nov 11, 2009 at 2:49 PM, Dan Nelson <dnelson at allantgroup.com> wrote:
>> > In the last episode (Nov 11), Ivan Voras said:
>> >> Sam Fourman Jr. wrote:
>> >> > I am running FreeBSD 8.0RC2 and I dont understand why my ZFS/NFS is
>> >> > acting weird on writes.   I get ~150mbit writes idk if this is good
>> >> > or not?   but it paused for a few seconds every once and awhile.
>> >>
>> >> You didn't give any "iostat" statistics - I suspect that if you
>> >> correlate ifstat and iostat output that you will see that network
>> >> "pauses" happen during spikes in IO.  You should check for this and
>> >> post your results.
>> >
>> > Yes, iostat would be useful here.  "iostat -zxC 2" will give you
>> > per-disk stats plus CPU usage every 2 seconds (CPU may be a factor if
>> > you have compression enabled).
>> >
>> > On a Solaris box I admin, setting zfs_write_limit_override helped
>> > stuttering while doing heavy writes.   It's not exported on FreeBSD, but
>> > it should be easy to add it as a RW sysctl; it lives in dsl_pool.c and
>> > can be tweaked at runtime.   Start big and tune it down so each write
>> > burst takes under a second; it looks like you're writing solid for
>> > around 6-8 seconds now.   The number will vary depending on your disk
>> > speed and how much ARC you have.
>>
>> here are some iostats for you. I do not believe I have compression enabled
>> am I mistaken?  isn'y SATA2 300MB/s?  and I am doing ~6MB/s per disk?  I
>> built this machine with 4GB of memory because I thought ZFS would like it.
>> now maybe a re(4) interface isnt the best choice.  if that is the problem
>> here I can change it.  We spent ~$800 on a hardware RAID card thinking
>> that it would help performance
>>
>> Why is it that with sftp we do not see the pauses in Network transfer?
>
> Your service times below are worrying (both for the NFS and sftp cases);
> anything above 50ms for a sustained period is usually a problem.  You
> mention hardware RAID, but I see 6 da# devices; are these just hooked up in
> passthrough mode, or is each da device backed by multiple SATA disks?  What
> kind of write caching do you have enabled on the RAID?  Can you disable it?
> It sort of looks like zfs is bursting more data to disk than the RAID card
> has RAM for, and it's spending multiple seconds just trying to recover.
> It's also odd that you're seeing queue depths up to 70 on those disks when
> zfs by default should only do 35 (sysctl vfs.zfs.vdev.max_pending).
>
> NFS has data consistency guarantees that require it to flush to disk more
> often than your sftp connection, so that may explain why the disk behaviour
> is different.
>


here is some sysctl output (i believe my max pending is 35)

# sysctl vfs.zfs
vfs.zfs.arc_meta_limit: 67108864
vfs.zfs.arc_meta_used: 26956528
vfs.zfs.mdcomp_disable: 0
vfs.zfs.arc_min: 33554432
vfs.zfs.arc_max: 268435456
vfs.zfs.zfetch.array_rd_sz: 1048576
vfs.zfs.zfetch.block_cap: 256
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8
vfs.zfs.prefetch_disable: 1
vfs.zfs.recover: 0
vfs.zfs.txg.synctime: 5
vfs.zfs.txg.timeout: 30
vfs.zfs.scrub_limit: 10
vfs.zfs.vdev.cache.bshift: 16
vfs.zfs.vdev.cache.size: 10485760
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.vdev.aggregation_limit: 131072
vfs.zfs.vdev.ramp_rate: 2
vfs.zfs.vdev.time_shift: 6
vfs.zfs.vdev.min_pending: 4
vfs.zfs.vdev.max_pending: 35
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_disable: 0
vfs.zfs.version.zpl: 3
vfs.zfs.version.vdev_boot: 1
vfs.zfs.version.spa: 13
vfs.zfs.version.dmu_backup_stream: 1
vfs.zfs.version.dmu_backup_header: 2
vfs.zfs.version.acl: 1
vfs.zfs.debug: 0
vfs.zfs.super_owner: 0

I am using this ARECA ARC-1130 in JBOD Mode
We have write cache disabled on the controller.

arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
> mem 0xfebff000-0xfebfffff,0xfdc00000-0xfdffffff irq 22 at device 14.0 on pci4
ARECA RAID ADAPTER0: Driver Version 1.20.00.16 2009-10-10
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.47 2009-06-25
arcmsr0: [ITHREAD]

The motherboard in this machine is Asus M4A785-M

http://usa.asus.com/product.aspx?P_ID=ef0qgvMIwOUagAVl&templete=2



if my hardware setup is the trouble, do I just need to start from
scratch and select and build a better machine?
in the end I was looking to get the fastest writes I can. I am just
confused on where I went wrong with this machine.

Sam Fourman Jr.