ZFS, Zvol, iSCSI and windows
Karli Sjöberg
karli.sjoberg at slu.se
Sat Sep 26 20:31:02 UTC 2015
Den 26 sep 2015 9:03 em skrev Willem Jan Withagen <wjw at digiware.nl>:
>
> On 25-9-2015 16:05, Tom Curry wrote:
> > On Fri, Sep 25, 2015 at 7:13 AM, Karli Sjöberg <karli.sjoberg at slu.se
> > <mailto:karli.sjoberg at slu.se>> wrote:
> >
> > fre 2015-09-25 klockan 11:08 +0200 skrev Willem Jan Withagen:
> > > Hi,
> > >
> > > Because of the Network Video Recorder (on windows) we use only likes
> > > "real" disks, and not SMB disks. We started using ZVOLs which are
> > > exported thru iSCSI/ctld....
> > >
> > > And that works really well, so there all thumbs up for this combo.
> > >
> > > However..... (you knew that was coming)
> > > I do have some questions, and hope somebody can share some insights.
> > >
> > > 1)
> > > This is a ZFS question.
> > >
> > > I have created a ZVOL with:
> > > zfs create -s -V 5T zfsraid/nvr2
> > >
> > > Looking at the disk usage in 'zfs get all':
> > > NAME PROPERTY VALUE SOURCE
> > > zfsraid/nvr2 used 9.97T -
> > > zfsraid/nvr2 available 438G -
> > > zfsraid/nvr2 referenced 9.97T -
> > > zfsraid/nvr2 compressratio 1.00x -
> > > zfsraid/nvr2 reservation none default
> > > zfsraid/nvr2 volsize 5T local
> > > zfsraid/nvr2 volblocksize 8K -
> > > zfsraid/nvr2 checksum on default
> > > zfsraid/nvr2 compression lz4 default
> > > zfsraid/nvr2 primarycache all default
> > > zfsraid/nvr2 secondarycache all default
> > > zfsraid/nvr2 usedbysnapshots 0 -
> > > zfsraid/nvr2 usedbydataset 9.97T -
> > > zfsraid/nvr2 usedbychildren 0 -
> > > zfsraid/nvr2 usedbyrefreservation 0 -
> > > zfsraid/nvr2 sync standard default
> > > zfsraid/nvr2 written 9.97T -
> > > zfsraid/nvr2 logicalused 4.97T -
> > > zfsraid/nvr2 logicalreferenced 4.97T -
> > > zfsraid/nvr2 volmode default default
> > >
> > > And what sort of "worries" me is that it seems that this volume is
> > using
> > > twice the amount of diskspace it is offering as ZVOL?
> > >
> > > a) Is this really true?
> > > b) Should I have done something different to not waste so much
> > overhead?
> > >
> > > Note that the compression rate is 1.00x, which is of course due to
> > > writing h264 media streams that do not compress at all. But is the
> > > parents default, and I haven't turned it off.
> > >
> > > 2)
> > > The second one might be more a Windows question, but anyways.
> > >
> > > The export is that same 5T ZVOL, plain create with
> > > zfs create -s -V 5T zfsraid/nvr2
> > > Under Windows I used the regular stuff to format the dis with GPT and
> > > NTFS with 8k segments. (matching the ZVOL blocksize)
> > >
> > > Upon reboot FreeBSD notices the following:
> > > GEOM_PART: partition 1 on (zvol/zfsraid/nvr2, GPT) is not aligned on
> > > 8192 bytes
> > > By itself not technical problem. But the warning does hint that the
> > > alignment is thus, that performance might suffer. the backing
> > disks are
> > > WD REDs which are 4K sectors.... So the misalignment could cause too
> > > many read/writes for writing.
> > >
> > > Does anybody have a clue as how to get Windows to do the aligning
> > correctly?
> >
> >
> > Is it this effect you are seeing:
> > https://forums.freebsd.org/threads/mitigating-4k-disk-issues-on-raidz.37365/
>
> The article attributes certain issues to using 4k blocks zvols in zraid.
> It could very well be on the money. But it seems testing is the best way
> to find out.
It may have been unclear but what I was trying to show in my posts was that while used space from the initiator point of view is always the same, used space in the storage differs wildly depending on the "volblocksize" you create the zvols with and the "Allocation unit size" you choose when you format the volume within the initiator. Seems as if you use a too small size, ZFS can't stripe the data across the vdev and is forced to write out the same data to all drives in the vdev. At least I hope the problem is confined to the vdev, but could be it's pool-wide (yikes!)
Anyhow, choosing a larger block- and allocation (cluster) size seems to mitigate it pretty efficiently, seems you actually didn't specify that when creating the zvol. Could you do "zfs get volblocksize zfsraid/nvr2" to see what was chosen? 8k? At least that seems to be the cluster size in the filesystem, judging by the output you posted below. I'd suggest you to make another one with 64k volblocksize/allocation unit size and see how that treets you, especially since you're dominantly storing large files, you've the least amount of overhead by doing so.
/K
>
> >
> > I ran into the same problem, the solution was to create the zvol with a
> > larger volblocksize like 32k. If space efficiency is the primary concern
> > I believe you need to choose a blocksize that is greater than or equal
> > to the number of non parity drives multiplied by their sector size.
> > Judging by your overhead I'm going to assume you have a 6 disk raidz2,
> > so 4 * 4k = 16k at least. Try creating another zvol with "-o
> > volblocksize=16k" and see how that treats you.
>
> Right guess,
>
> Actually the it is a 2* 6 vol raidz2
>
> THe article seems to show that the overhead gets less and less when the
> blocksize increases.
> In which case I start worrying about the performance, because writting
> smaller blocks will require a read/wite cycle.
>
> In my specific case I'm writing large video files (5Gb) in streaming
> mode, so the read/write effect will be mitigated due to streaming.
>
> I'll go and make a few ZVOLS and see what happens with the overhead when
> they start filling up. But it looks like the windows blocksize should
> certainly not match zvol blocksize.
>
> > As to your second question, modern versions of windows should align
> > partitions correctly, what version are you using? You may wish to open
> > an elevated command prompt and sanity check the output of "fsutil fsinfo
> > ntfsinfo x:"
>
> I'm just loading the disk in explorer, rightclick it, and then format.
> Just as basic as that.
> Not sure what to make of it, but this gives:
>
> C:\WINDOWS\system32>fsutil fsinfo ntfsinfo m:
> NTFS Volume Serial Number : 0x2aaa50b8aa50826d
> NTFS Version : 3.1
> LFS Version : 2.0
> Number Sectors : 0x000000027ffbefff
> Total Clusters : 0x0000000027ffbeff
> Free Clusters : 0x00000000008497fc
> Total Reserved : 0x00000000000a8ec0
> Bytes Per Sector : 512
> Bytes Per Physical Sector : 4096
> Bytes Per Cluster : 8192
> Bytes Per FileRecord Segment : 1024
> Clusters Per FileRecord Segment : 0
> Mft Valid Data Length : 0x0000000000140000
> Mft Start Lcn : 0x0000000000060000
> Mft2 Start Lcn : 0x0000000000000001
> Mft Zone Start : 0x00000000000600a0
> Mft Zone End : 0x0000000000066420
> Max Device Trim Extent Count : 4294967295
> Max Device Trim Byte Count : 0xffffffff
> Max Volume Trim Extent Count : 62
> Max Volume Trim Byte Count : 0x40000000
> Resource Manager Identifier : FD836E1F-1372-11E5-825F-000272131F3C
>
> The gpart info in the zvol is reported as unaligned when booting
> FreeBSD, but that would be because windows put it in there in an
> unaligned postition. Something like at 63 * 512 bytes...??
>
> Any commandline incarnation that allows me to put it a a different offset?
>
> >
>
> > Finally, from my own experience with using iscsi under windows I
> > found I had a random 4k iops increase from ~10,000 to ~90,000 (which
> > is close to my pools real speed) by adjusting the following registry
> > key
>
>
> > Create this DWORD key with a value of 1:
> > HKLM\SYSTEM\CurrentControlSet\Control\Class\{4D36E97B-E325-11CE-BFC1-08002BE10318}\<Instance
> > Number>\Parameters\iSCSIDisableNagle
>
> Very usefull hint, thanx for that.
>
> --WjW
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
More information about the freebsd-fs
mailing list