[ZFS] refquota is very slow !

[Ben RUBSON]

OK !
Well, Juergen you put me on the right way.
I investigated further and further and found that refquota slowness is not due to the number of files but to the remaining free space.
Let me explain this.

Let's create a pool
# zpool create -O atime=off test /dev/...

And do some fio tests with quota on it :
# zfs set quota=500M refquota=none test/test ; cd /test/test
# fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=499M
  WRITE: io=510976KB, aggrb=805955KB/s, minb=805955KB/s, maxb=805955KB/s, mint=634msec, maxt=634msec

OK, very fast (nice SSD), even if we have filled the space up to the quota.

Now, let's do some fio tests with refquota :
# zfs set quota=none refquota=500M test/test ; cd /test/test
# fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=499M
  WRITE: io=510976KB, aggrb=1679KB/s, minb=1679KB/s, maxb=1679KB/s, mint=304177msec, maxt=304177msec (!!!)
# fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=490M
  WRITE: io=501760KB, aggrb=44010KB/s, minb=44010KB/s, maxb=44010KB/s, mint=11401msec, maxt=11401msec
# fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=480M
  WRITE: io=491520KB, aggrb=92774KB/s, minb=92774KB/s, maxb=92774KB/s, mint=5298msec, maxt=5298msec
# fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=450M
  WRITE: io=460800KB, aggrb=169038KB/s, minb=169038KB/s, maxb=169038KB/s, mint=2726msec, maxt=2726msec
# fio --bs=8192 --rw=write --iodepth=24 --name=test --filename=test.bin --direct=1 --ioengine=sync --size=400M
  WRITE: io=409600KB, aggrb=215126KB/s, minb=215126KB/s, maxb=215126KB/s, mint=1904msec, maxt=1904msec

So, refquota is very very slow when filled space is near the limit, as if it had some (huge) overhead for each block written.

Now let's do some filling tests with dd, first with quota :
# zfs set quota=500M refquota=none test/test
# dd if=/dev/random of=/test/test/bf conv=sync bs=64k
523698176 bytes transferred in 6.411629 secs (81679430 bytes/sec)
# dd if=/dev/random of=/test/test/bf conv=sync bs=8K
523640832 bytes transferred in 6.846064 secs (76487868 bytes/sec)
# dd if=/dev/random of=/test/test/bf conv=sync bs=4K
523636736 bytes transferred in 7.333179 secs (71406514 bytes/sec)
# dd if=/dev/random of=/test/test/bf conv=sync bs=1K
523633664 bytes transferred in 10.312721 secs (50775511 bytes/sec)
# dd if=/dev/random of=/test/test/bf conv=sync bs=512
523633152 bytes transferred in 14.357943 secs (36469928 bytes/sec)

And now the same filling tests with refquota :
# zfs set quota=none refquota=500M test/test
# dd if=/dev/random of=/test/test/bf conv=sync bs=64k
523698176 bytes transferred in 6.440955 secs (81307534 bytes/sec)
# dd if=/dev/random of=/test/test/bf conv=sync bs=8K
523640832 bytes transferred in 10.264528 secs (51014602 bytes/sec)
# dd if=/dev/random of=/test/test/bf conv=sync bs=4K
523636736 bytes transferred in 14.150774 secs (37004106 bytes/sec)
# dd if=/dev/random of=/test/test/bf conv=sync bs=1K
523633664 bytes transferred in 40.314732 secs (12988643 bytes/sec)
# dd if=/dev/random of=/test/test/bf conv=sync bs=512
523633152 bytes transferred in 74.449636 secs (7033388 bytes/sec)

OK so here, filling remaining space with small blocks, there are of course more write operations than with big blocks, refquota overhead is then really impacting. 

I think that all these tests reveal that refquota has some overhead compared to quota to compute a block write when used space is near the limit.
Do you agree ?

Question would be then, why this overhead, and could it be reduced (as with quota) ?
(certainly a dev question here)

Thank you again,

Ben