ZFS/RAIDZ and SAMBA: abyssimal performance
ml at my.gd
Fri Jan 4 14:06:05 UTC 2013
On Jan 4, 2013, at 2:52 PM, "O. Hartmann" <ohartman at zedat.fu-berlin.de> wrote:
> I use a small testing server. The hardware is most modern Intel hardware
> (i3-3220, Z77 chipset), 16GB RAM. The OS is FreeBSD 10.0-CURRENT #1
> r245036M: Fri Jan 4 12:48:53 CET 2013.
> The ZFS subsystem is comprised by 3 Western Digital 3 TB harddrives (WDC
> WD30EZRX-00DC0B0 80.00A80> ATA-9 SATA 3.x device), setup as a ZFS RAIDZ:
> root at gate [etc] zpool status
> pool: ASGARD00
> state: ONLINE
> scan: scrub repaired 0 in 1h45m with 0 errors on Sat Dec 1 20:59:44 2012
> NAME STATE READ
> WRITE CKSUM
> ASGARD00 ONLINE 0
> 0 0
> raidz1-0 ONLINE 0
> 0 0
> gptid/1e716118-1492-11e2-b828-90f6526a24d6 ONLINE 0
> 0 0
> gptid/294a6798-1492-11e2-b828-90f6526a24d6 ONLINE 0
> 0 0
> gptid/30c813f8-1492-11e2-b828-90f6526a24d6 ONLINE 0
> 0 0
> ada0p1 ONLINE 0
> 0 0
> ada0p2 ONLINE 0
> 0 0
> errors: No known data errors
> The "logs" and "cache" device is a single SAMSUNG 830 SSD, 60 GB
> capacity, GPT partinioned, logs (ZIL) has 5GB, cache has ~55 GB.
> I think its not the optimal setup using the very same SSD for both
> caching/L2ARC and ZIL, but without the cache device the performance
> doen't differ much at the moment. Luckliy, with ZFS I can change the
> arrangement as I like.
> The ZFS volumes created on the pool named ASGARD00 are standard, only
> options sharenfs/sharesmb/checksum are set to yes. Everthing elese is
> set to the defaults.
> In /boot/loader.conf I set the following parameters according to many
> (and confusing!) help and suggestions on the web:
> # ZFS
> #vfs.zfs.write_limit_override=1073741824 # 1GB
> The NFSv4 performance (client is also FreeBSD 10.0-CURRENT of the same
> date) is moderate to disapointing and doesn't exceed 45 - 55 MB/s
> sustained, but here are sometimes "spikes" I can watch with "systat -vm
> 1" reporting 120 MB/s per drive (ada2/ada3/ada4, the 3x 3TB WD drives in
> RAIDZ). I still benchmark via iozone. Both server and client use JUMBO
> frames (MTU=6120), which gives better throughput compared to the
> standard MTU=1500.
> The local performance on the server itself is slightly better, but
> iozone reports some strange numbers. The benchmark "writes" (using 4
> threads, 4k blocksizes, writing four times files of size 1G to the ZFS
> volume reports sometimes 150 MB/s throughput, and then 70 MB/s and
> re-writes is then 1/10 of the "write" throughput and according to the
> manual of iozone, re-write is considered to have higher values due to
> the lack of writing the meta data again. But I'm still testing this case.
> Well, the ZFS volumes are also shared as SAMBA CIFS volumes and here I
> experience something that is simply described as "abyssimal"
> performance! From both a dedicated Windows 7 Pro client and a VirtualBox
> 4.2.6-client access to folders in a share, say my local home, can take
> ages! Opening files takes eons, if possible, in most cases windows
> reports "can not open ...". Copying files from Windows to the SAMBA
> share doesn't work or take ages, the throughput visible on the server
> side watched by "systat -vm 1" reports spiking 0.48 MB/s, with a hiatus
> of several seconds.
> Well, the SAMBA setup is straightforward, for two weeks now I have
> permutated nearly every parameter suggested on all the web's help sites
> and I simply took the well configuration from one of our lab's FreeBSD
> 9.1-STABLE SAMBA servers and changed the local settings for IP and
> domain names etc. The working server (FreeBSD 9.1-STABLE) in question
> has a single ZFS drive and is exporting this also via NFSv4. It doesn't
> have RAIDZ setup!
> Before I start benchmarking further with iozone I need to know whether
> there is an unresolved problem in FreeBSD 10.0 with ZFS/RAIDZ and SAMBA
> or whether I'm mislead and have overseen an important setup option.
> Before exposing all of my setups here I need to clearify.
> I didn't find so far any issues on the web regarding SAMBA, NFSv4 and
> Thanks in advance,
I experienced the same performance problem, then followed Jeremy Chadwick's advice and tuned variables a bit, I'm getting excellent performance now.
# Tune ZFS somewhat aye ?
# Decrease ZFS txg timeout value from 30 (default) to 5 seconds. This
# should increase throughput and decrease the "bursty" stalls that
# happen during immense I/O with ZFS.
And network cards:
# Up a bit our intel cards parameters
And this is under [global] in /usr/local/etc/smb.conf:
min receivefile size = 16384
aio read size = 16384
aio write size = 16384
aio write behind = yes
More information about the freebsd-current