[ZFS] refquota is very slow !

[InterNetX - Juergen Gotteswinter]

> Ben RUBSON <ben.rubson at gmail.com> hat am 7. September 2016 um 10:44
> geschrieben:
>
>
>
> OK !
> Well, Juergen you put me on the right way.
> I investigated further and further and found that refquota slowness is not due
> to the number of files but to the remaining free space.
> Let me explain this.
>
> Let's create a pool
> # zpool create -O atime=off test /dev/...
>
> And do some fio tests with quota on it :
> # zfs set quota=500M refquota=none test/test ; cd /test/test
> # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin
> --direct=1 --ioengine=sync --size=499M
> WRITE: io=510976KB, aggrb=805955KB/s, minb=805955KB/s, maxb=805955KB/s,
> mint=634msec, maxt=634msec
>
> OK, very fast (nice SSD), even if we have filled the space up to the quota.
>
> Now, let's do some fio tests with refquota :
> # zfs set quota=none refquota=500M test/test ; cd /test/test
> # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin
> --direct=1 --ioengine=sync --size=499M
> WRITE: io=510976KB, aggrb=1679KB/s, minb=1679KB/s, maxb=1679KB/s,
> mint=304177msec, maxt=304177msec (!!!)
> # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin
> --direct=1 --ioengine=sync --size=490M
> WRITE: io=501760KB, aggrb=44010KB/s, minb=44010KB/s, maxb=44010KB/s,
> mint=11401msec, maxt=11401msec
> # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin
> --direct=1 --ioengine=sync --size=480M
> WRITE: io=491520KB, aggrb=92774KB/s, minb=92774KB/s, maxb=92774KB/s,
> mint=5298msec, maxt=5298msec
> # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin
> --direct=1 --ioengine=sync --size=450M
> WRITE: io=460800KB, aggrb=169038KB/s, minb=169038KB/s, maxb=169038KB/s,
> mint=2726msec, maxt=2726msec
> # fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin
> --direct=1 --ioengine=sync --size=400M
> WRITE: io=409600KB, aggrb=215126KB/s, minb=215126KB/s, maxb=215126KB/s,
> mint=1904msec, maxt=1904msec
>
> So, refquota is very very slow when filled space is near the limit, as if it
> had some (huge) overhead for each block written.
>
> Now let's do some filling tests with dd, first with quota :
> # zfs set quota=500M refquota=none test/test
> # dd if=/dev/random of=/test/test/bf conv=sync bs=64k
> 523698176 bytes transferred in 6.411629 secs (81679430 bytes/sec)
> # dd if=/dev/random of=/test/test/bf conv=sync bs=8K
> 523640832 bytes transferred in 6.846064 secs (76487868 bytes/sec)
> # dd if=/dev/random of=/test/test/bf conv=sync bs=4K
> 523636736 bytes transferred in 7.333179 secs (71406514 bytes/sec)
> # dd if=/dev/random of=/test/test/bf conv=sync bs=1K
> 523633664 bytes transferred in 10.312721 secs (50775511 bytes/sec)
> # dd if=/dev/random of=/test/test/bf conv=sync bs=512
> 523633152 bytes transferred in 14.357943 secs (36469928 bytes/sec)
>
> And now the same filling tests with refquota :
> # zfs set quota=none refquota=500M test/test
> # dd if=/dev/random of=/test/test/bf conv=sync bs=64k
> 523698176 bytes transferred in 6.440955 secs (81307534 bytes/sec)
> # dd if=/dev/random of=/test/test/bf conv=sync bs=8K
> 523640832 bytes transferred in 10.264528 secs (51014602 bytes/sec)
> # dd if=/dev/random of=/test/test/bf conv=sync bs=4K
> 523636736 bytes transferred in 14.150774 secs (37004106 bytes/sec)
> # dd if=/dev/random of=/test/test/bf conv=sync bs=1K
> 523633664 bytes transferred in 40.314732 secs (12988643 bytes/sec)
> # dd if=/dev/random of=/test/test/bf conv=sync bs=512
> 523633152 bytes transferred in 74.449636 secs (7033388 bytes/sec)
>
> OK so here, filling remaining space with small blocks, there are of course
> more write operations than with big blocks, refquota overhead is then really
> impacting.
 
 
thats a general problem which you will also see on nearly full pools, try to no
exceed 90% better 80-85%, its the nature of cow fs that this effect occours

>
> I think that all these tests reveal that refquota has some overhead compared
> to quota to compute a block write when used space is near the limit.
> Do you agree ?
>
> Question would be then, why this overhead, and could it be reduced (as with
> quota) ?
> (certainly a dev question here)
>
> Thank you again,
>
> Ben
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"