ZFS, Zvol, iSCSI and windows
Willem Jan Withagen
wjw at digiware.nl
Fri Oct 2 08:24:46 UTC 2015
On 26-9-2015 21:03, Willem Jan Withagen wrote:
> On 25-9-2015 16:05, Tom Curry wrote:
>> On Fri, Sep 25, 2015 at 7:13 AM, Karli Sjöberg <karli.sjoberg at slu.se
>> <mailto:karli.sjoberg at slu.se>> wrote:
>>
>> fre 2015-09-25 klockan 11:08 +0200 skrev Willem Jan Withagen:
>> > Hi,
>> >
>> > Because of the Network Video Recorder (on windows) we use only likes
>> > "real" disks, and not SMB disks. We started using ZVOLs which are
>> > exported thru iSCSI/ctld....
>> >
>> > And that works really well, so there all thumbs up for this combo.
>> >
>> > However..... (you knew that was coming)
>> > I do have some questions, and hope somebody can share some insights.
>> >
>> > 1)
>> > This is a ZFS question.
>> >
>> > I have created a ZVOL with:
>> > zfs create -s -V 5T zfsraid/nvr2
>> >
>> > Looking at the disk usage in 'zfs get all':
>> > NAME PROPERTY VALUE SOURCE
>> > zfsraid/nvr2 used 9.97T -
>> > zfsraid/nvr2 available 438G -
>> > zfsraid/nvr2 referenced 9.97T -
>> > zfsraid/nvr2 compressratio 1.00x -
>> > zfsraid/nvr2 reservation none default
>> > zfsraid/nvr2 volsize 5T local
>> > zfsraid/nvr2 volblocksize 8K -
>> > zfsraid/nvr2 checksum on default
>> > zfsraid/nvr2 compression lz4 default
>> > zfsraid/nvr2 primarycache all default
>> > zfsraid/nvr2 secondarycache all default
>> > zfsraid/nvr2 usedbysnapshots 0 -
>> > zfsraid/nvr2 usedbydataset 9.97T -
>> > zfsraid/nvr2 usedbychildren 0 -
>> > zfsraid/nvr2 usedbyrefreservation 0 -
>> > zfsraid/nvr2 sync standard default
>> > zfsraid/nvr2 written 9.97T -
>> > zfsraid/nvr2 logicalused 4.97T -
>> > zfsraid/nvr2 logicalreferenced 4.97T -
>> > zfsraid/nvr2 volmode default default
>> >
>> > And what sort of "worries" me is that it seems that this volume is
>> using
>> > twice the amount of diskspace it is offering as ZVOL?
>> >
>> > a) Is this really true?
>> > b) Should I have done something different to not waste so much
>> overhead?
>> >
>> > Note that the compression rate is 1.00x, which is of course due to
>> > writing h264 media streams that do not compress at all. But is the
>> > parents default, and I haven't turned it off.
>> >
>> > 2)
>> > The second one might be more a Windows question, but anyways.
>> >
>> > The export is that same 5T ZVOL, plain create with
>> > zfs create -s -V 5T zfsraid/nvr2
>> > Under Windows I used the regular stuff to format the dis with GPT and
>> > NTFS with 8k segments. (matching the ZVOL blocksize)
>> >
>> > Upon reboot FreeBSD notices the following:
>> > GEOM_PART: partition 1 on (zvol/zfsraid/nvr2, GPT) is not aligned on
>> > 8192 bytes
>> > By itself not technical problem. But the warning does hint that the
>> > alignment is thus, that performance might suffer. the backing
>> disks are
>> > WD REDs which are 4K sectors.... So the misalignment could cause too
>> > many read/writes for writing.
>> >
>> > Does anybody have a clue as how to get Windows to do the aligning
>> correctly?
>>
>>
>> Is it this effect you are seeing:
>> https://forums.freebsd.org/threads/mitigating-4k-disk-issues-on-raidz.37365/
>
> The article attributes certain issues to using 4k blocks zvols in zraid.
> It could very well be on the money. But it seems testing is the best way
> to find out.
>
>>
>> I ran into the same problem, the solution was to create the zvol with a
>> larger volblocksize like 32k. If space efficiency is the primary concern
>> I believe you need to choose a blocksize that is greater than or equal
>> to the number of non parity drives multiplied by their sector size.
>> Judging by your overhead I'm going to assume you have a 6 disk raidz2,
>> so 4 * 4k = 16k at least. Try creating another zvol with "-o
>> volblocksize=16k" and see how that treats you.
>
> Right guess,
>
> Actually the it is a 2* 6 vol raidz2
>
> THe article seems to show that the overhead gets less and less when the
> blocksize increases.
> In which case I start worrying about the performance, because writting
> smaller blocks will require a read/wite cycle.
>
> In my specific case I'm writing large video files (5Gb) in streaming
> mode, so the read/write effect will be mitigated due to streaming.
>
> I'll go and make a few ZVOLS and see what happens with the overhead when
> they start filling up. But it looks like the windows blocksize should
> certainly not match zvol blocksize.
>
>> As to your second question, modern versions of windows should align
>> partitions correctly, what version are you using? You may wish to open
>> an elevated command prompt and sanity check the output of "fsutil fsinfo
>> ntfsinfo x:"
>
> I'm just loading the disk in explorer, rightclick it, and then format.
> Just as basic as that.
> Not sure what to make of it, but this gives:
>
> C:\WINDOWS\system32>fsutil fsinfo ntfsinfo m:
> NTFS Volume Serial Number : 0x2aaa50b8aa50826d
> NTFS Version : 3.1
> LFS Version : 2.0
> Number Sectors : 0x000000027ffbefff
> Total Clusters : 0x0000000027ffbeff
> Free Clusters : 0x00000000008497fc
> Total Reserved : 0x00000000000a8ec0
> Bytes Per Sector : 512
> Bytes Per Physical Sector : 4096
> Bytes Per Cluster : 8192
> Bytes Per FileRecord Segment : 1024
> Clusters Per FileRecord Segment : 0
> Mft Valid Data Length : 0x0000000000140000
> Mft Start Lcn : 0x0000000000060000
> Mft2 Start Lcn : 0x0000000000000001
> Mft Zone Start : 0x00000000000600a0
> Mft Zone End : 0x0000000000066420
> Max Device Trim Extent Count : 4294967295
> Max Device Trim Byte Count : 0xffffffff
> Max Volume Trim Extent Count : 62
> Max Volume Trim Byte Count : 0x40000000
> Resource Manager Identifier : FD836E1F-1372-11E5-825F-000272131F3C
>
> The gpart info in the zvol is reported as unaligned when booting
> FreeBSD, but that would be because windows put it in there in an
> unaligned postition. Something like at 63 * 512 bytes...??
>
> Any commandline incarnation that allows me to put it a a different offset?
>
>>
>
>> Finally, from my own experience with using iscsi under windows I
>> found I had a random 4k iops increase from ~10,000 to ~90,000 (which
>> is close to my pools real speed) by adjusting the following registry
>> key
>
>
>> Create this DWORD key with a value of 1:
>> HKLM\SYSTEM\CurrentControlSet\Control\Class\{4D36E97B-E325-11CE-BFC1-08002BE10318}\<Instance
>> Number>\Parameters\iSCSIDisableNagle
>
> Very usefull hint, thanx for that.
And as a followup, I created a new ZVOL:
zfs create -V 2T -b 128k -o compression=off zfsraid/nvr
That gives:
NAME PROPERTY VALUE SOURCE
zfsraid/nvr used 2.00T -
zfsraid/nvr referenced 1.86T -
zfsraid/nvr compressratio 1.00x -
zfsraid/nvr reservation none default
zfsraid/nvr volsize 2T local
zfsraid/nvr volblocksize 128K -
zfsraid/nvr checksum on default
zfsraid/nvr compression off local
zfsraid/nvr refreservation 2.00T local
zfsraid/nvr usedbysnapshots 0 -
zfsraid/nvr usedbydataset 1.86T -
zfsraid/nvr usedbychildren 0 -
zfsraid/nvr usedbyrefreservation 150G -
zfsraid/nvr written 1.86T -
zfsraid/nvr logicalused 1.86T -
zfsraid/nvr logicalreferenced 1.86T -
And if there is any overhead, it does not really show here.
So the moral of the story:
It is not always wise to match NTFS blocksize with the ZVOL blocksize.
As for possible performance penalties on the Windows side....
It is hard to find a niche and sensible disk tester.
Using the error scanner from HDtune, performance has dropped a tiny bit.
I'm seeing a 1000 IOPS and 30Mbyte/sec of throuput.
This is on a 2* vdev 6* disk raidz2 volume.
--WjW
More information about the freebsd-fs
mailing list