samba+zfs
Kurt Touet
ktouet at gmail.com
Wed Nov 9 23:12:34 UTC 2011
On Wed, Nov 9, 2011 at 1:39 PM, Garrett Cooper <yanegomi at gmail.com> wrote:
> On Nov 8, 2011, at 11:07 PM, "Daniel O'Connor" <doconnor at gsoft.com.au> wrote:
>
>>
>> On 09/11/2011, at 17:32, Garrett Cooper wrote
>>>> dd's of large files (spooled backups going to tape) to /dev/null are as slow as Samba.
>>>
>>> - Dedupe?
>>
>> Nope.
>>
>>> - Compression?
>>
>> On the mail spool & ports, but not on the tape spool.
>>
>>> - How much RAM?
>>
>> 8GB.
>>> - What debug options do you have enabled in the kernel?
>>
>> It is 8.2-GENERIC so.. no WITNESS (for example)
>
> Ok. 8.2 release or stable?
>
>>> I've been noticing a slowdown in some respects with NFS/SMB, but I
>>> suspected it was because I have an re(4) based NIC. ZFS has also wired
>>> down a lot of my system memory for the L2ARC…
>>
>>
>> re isn't great but I wouldn't expect it to slow down over time.. Unless bounce buffers got used more and more or something.
>>
>> I have an em0 card in this system - but in any case it is slow locally (i.e. dd a large file with 64k block size).
>
> Good point. Simple base cases help isolate the root cause. That
> being said, my disk speed(s) are a lot better than my network + samba
> speeds (zfs:store is mfid0 backed with write cache enabled; zfs:sac is
> a single ada(4) backed disk with write cache enabled -- err... it
> shouldn't be like that), but I suspect that's misconfiguration on my
> part:
>
> $ sysctl hw.model hw.physmem
> hw.model: Intel(R) Xeon(R) CPU W3520 @ 2.67GHz
> hw.physmem: 12863094784
> $ sudo mfiutil show volumes
> mfi0 Volumes:
> Id Size Level Stripe State Cache Name
> mfid0 ( 1860G) RAID-6 64k OPTIMAL Enabled <STORE>
> $ zpool status
> pool: sac
> state: ONLINE
> scan: none requested
> config:
>
> NAME STATE READ WRITE CKSUM
> sac ONLINE 0 0 0
> ada0p3 ONLINE 0 0 0
>
> errors: No known data errors
>
> pool: store
> state: ONLINE
> scan: scrub repaired 0 in 10h9m with 0 errors on Mon Nov 7 18:58:01 2011
> config:
>
> NAME STATE READ WRITE CKSUM
> store ONLINE 0 0 0
> mfid0p1 ONLINE 0 0 0
>
> errors: No known data errors
> $ zfs list -o name,mountpoint,atime,sync,compression,dedup
> NAME MOUNTPOINT ATIME SYNC
> COMPRESS DEDUP
> sac legacy on standard
> off off
> sac/root / on standard
> off off
> sac/scratch /scratch on standard
> off off
> sac/scratch/freenas /scratch/freenas off standard
> on off
> sac/scratch/freenas/FreeBSD /scratch/freenas/FreeBSD off standard
> on off
> sac/usr /usr on standard
> off off
> sac/var /var on standard
> off off
> store /store on standard
> off off
> store/freebsd /store/freebsd on standard
> on on
> store/home /usr/home on standard
> off off
> $ dd if=/dev/zero of=foo bs=1m count=1024
> 1024+0 records out
> 1073741824 bytes transferred in 13.426620 secs (79971119 bytes/sec)
> $ cd /store
> $ dd if=/dev/zero of=foo bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 7.565117 secs (141933276 bytes/sec)
> $ cat /usr/local/etc/smb.conf
> [global]
> workgroup = WORKGROUP
> server string = BAYONETTA
> security = user
> load printers = no
> max log size = 50
> preferred master = yes
> local master = yes
> socket options = SO_RCVBUF=16384 SO_SNDBUF=16384
> nt acl support = yes
> inherit acls = yes
> map acl inherit = yes
> aio read size = 16384
> aio write size = 16384
>
> [scratch]
> path = /scratch
> writeable = yes
>
> [store]
> path = /store
> writeable = yes
> $
>
> I'll have to:
> 1. Recheck what Windows 7 says when transferring out to my box
> with a large file.
> 2. Use nc to quickly measure network performance.
> 3. Try transferring over NFS with both my Macbook and setup
> FreeBSD or Linux on the other workstation for testing out NFS
> transfers (64kB rsize/wsize of course). Wash, rinse, repeat with
> samba.
> The last I remember the transfer speeds were pitiful with samba36
> (somewhere around 45MBps to my 'store' zpool). I've been conservative
> with the socket settings, but it might be time to bump that up along
> with the mbuf cluster count (for some odd reason I haven't changed it
> from the system default), reboot, and retest.
> Thanks,
> -Garrett
>
I have found the source of my speed issues, and they are not tied to
samba or zfs. The replacement drive that I put into my array (just
after I updated to stable/9 a few weeks ago) is hitting 100% busy in
gstat while only writing out1-3MB/s -- it's a WDC WD15EARS-00Z5B1 and
some quick googling has demonstrated that there are issues with drives
like it.
More information about the freebsd-current
mailing list