ZFS v28 and free space leakage
Peter Maloney
peter.maloney at brockmann-consult.de
Tue Apr 3 13:22:13 UTC 2012
On 04/03/2012 10:44 AM, Volodymyr Kostyrko wrote:
> Peter Maloney wrote:
>> To prove there is a leak, you would need to fill up the disk, delete
>> everything, and then fill it again to see if it fit less. If I did such
>> a test and it was the same, I would just forget about the problem.
>
> kk, throwing junk in:
>
> # zpool list
> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
> kohrah1 136G 22,4M 136G 0% 1.00x ONLINE -
> kohrah1new 136G 2,29G 134G 1% 1.00x ONLINE -
> # find /kohrah1new/ | wc -l
> 150590
> # rm -rf /kohrah1new/*
> # find /kohrah1new/
> /kohrah1new/
> # zpool list
> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
> kohrah1 136G 22,4M 136G 0% 1.00x ONLINE -
> kohrah1new 136G 436K 136G 0% 1.00x ONLINE -
>
> Not a same test you ask but it seems that ZFS leaks on metadata. Or
> peruses them. Repeating the same tasks results in:
Yes, my test was suggested so you would really fill it to the limit, and
then delete and refill. It was expected that when removing, the "ALLOC"
is high still, but the hypothesis was that it is just blank allocated
space for data that doesn't exist, and will be reused when it is filled
again. So to test what I wanted, you really need to fill it to the max
twice. I thought you would do this since your pool is only 136 GB.
So I decided to try it in a virtual machine, with a very small disk (700
MB slice). The result is attached. The conclusion is that the space is
not really missing... if anything is missing it is probably less than a
kilobyte after filling and removing all more than 11 times, probably 40
or 50 times.
>
> # zpool list
> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
> kohrah1 136G 22,4M 136G 0% 1.00x ONLINE -
> kohrah1new 136G 336K 136G 0% 1.00x ONLINE -
>
> So this feels like some leftover.
-------------- next part --------------
================================
zfs space leak experiment
================================
Prepare:
[root at bczfsvm1test /test]# uname -a
FreeBSD bczfsvm1test.bc.local 8.2-STABLE-20120204 FreeBSD 8.2-STABLE-20120204 #0: Mon Feb 6 12:10:32 UTC 2012 root at bczfsvm1.bc.local:/usr/obj/usr/src/sys/GENERIC amd64
[root at bczfsvm1test ~]# zpool create test da2
[root at bczfsvm1test ~]# zpool list test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 19.9G 78.5K 19.9G 0% 1.00x ONLINE -
[root at bczfsvm1test ~]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 91K 19.6G 31K /test
[root at bczfsvm1test ~]# cd /test
[root at bczfsvm1test ~]# chmod ugo=rwx .
==============================
test1a
==============================
[root at bczfsvm1test /test]# cp ~/FreeBSD-8.2-STABLE-20120204.iso 1
cp: 1: No space left on device
[root at bczfsvm1test /test]# ls -l
total 675176
-rw-r--r-- 1 root wheel 690880512 Feb 22 17:51 1
[root at bczfsvm1test /test]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
[root at bczfsvm1test /test]# rm 1
==============================
test1b
==============================
[root at bczfsvm1test /test]# cp /test/FreeBSD-8.2-STABLE-20120204.iso 2
cp: 2: No space left on device
[root at bczfsvm1test /test]# ls -l
total 675177
-rw-r--r-- 1 root wheel 690880512 Feb 22 17:52 2
[root at bczfsvm1test /test]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
[root at bczfsvm1test /test]# rm 2
==============================
test2a
==============================
[root at bczfsvm1test /test]# for a in {1..10}; do mkdir $a; done
[root at bczfsvm1test /test]# for a in {1..10}; do mkdir $a; for b in {1..5000}; do dd if=/dev/random of=$a/$b bs=128k count=10; done; done
...(runs long)
dd: 1/1227: No space left on device
[root at bczfsvm1test /test]# du -sxc *
675392 1
2 10
2 2
2 3
2 4
2 5
2 6
2 7
2 8
2 9
675405 total
[root at bczfsvm1test /test]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 188K 659M /test
[root at bczfsvm1test /test]# zpool list test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 32.1M 95% 1.00x ONLINE -
[root at bczfsvm1test /test]# rm -rf *
[root at bczfsvm1test /test]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 424K 660M 31K /test
[root at bczfsvm1test /test]# zpool list test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 460K 692M 0% 1.00x ONLINE -
==============================
test2b
==============================
[root at bczfsvm1test /test]# for a in {1..10}; do mkdir $a; for b in {1..5000}; do dd if=/dev/random of=$a/$b bs=128k count=10; done; done
...(runs long)
dd: 1/977: No space left on device
[root at bczfsvm1test /test]# du -sxc *
675391 1
675391 total
[root at bczfsvm1test /test]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
[root at bczfsvm1test /test]# zpool list test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
[root at bczfsvm1test /test]# rm -rf *
[root at bczfsvm1test /test]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 420K 660M 31K /test
[root at bczfsvm1test /test]# zpool list test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 456K 692M 0% 1.00x ONLINE -
==============================
test2c
==============================
[root at bczfsvm1test /test]# for a in {1..10}; do mkdir $a; for b in {1..5000}; do dd if=/dev/random of=$a/$b bs=128k count=10; done; done
...(runs long)
dd: 1/1192: No space left on device
[root at bczfsvm1test /test]# du -sxc *
675380 1
675380 total
[root at bczfsvm1test /test]# x=1; while true; do touch blah_$x; let x++; done
[root at bczfsvm1test /test]# du -sxc *
675380 1
675380 total
[root at bczfsvm1test /test]# rm -rf /test/*
(so far, it looks like a very small amount between 11 kB and 14 kB is lost every time)
==============================
test3
==============================
run=1
rm -rf /test/*
while true; do
echo "============="
echo "Run #$run"
full=0
for a in {1..10}; do
mkdir $a
for b in {1..500}; do
dd if=/dev/random of=$a/$b bs=128k count=10 >/dev/null 2>&1
if [ "$?" != 0 ]; then
full=1
fi
done
if [ "$full" = 1 ]; then
break
fi
done
echo "Full:"
du -sx /test
zfs list test
zpool list test
rm -rf /test/*
sleep 30
echo "Clear:"
zfs list test
zpool list test
let run++
done
old run with version 1 of the script (only du output):
675283 1
675283 total
675387 1
675387 total
675266 1
675266 total
675520 1
675520 total
675399 1
675399 total
675272 1
675272 total
675392 1
675392 total
675527 1
675527 total
675268 1
675268 total
675398 1
675398 total
run with version 2 (zfs list, etc. added):
=============
Run #1
Full:
675280 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 80.8M 579M 80.2M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 81.0M 611M 11% 1.00x ONLINE -
=============
Run #2
Full:
675547 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 660M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 661M 31.5M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 132M 528M 132M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 132M 560M 19% 1.00x ONLINE -
=============
Run #3
Full:
675398 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.7M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 59.6M 600M 58.9M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 59.8M 632M 8% 1.00x ONLINE -
=============
Run #4
Full:
675277 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 34.5M 625M 33.9M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 34.7M 657M 5% 1.00x ONLINE -
=============
Run #5
Full:
675391 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.7M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 502M 158M 502M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 502M 190M 72% 1.00x ONLINE -
=============
Run #6
Full:
675386 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.9M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 82.1M 578M 81.5M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 82.2M 610M 11% 1.00x ONLINE -
=============
Run #7
Full:
675395 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.6M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 116M 544M 115M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 116M 576M 16% 1.00x ONLINE -
=============
Run #8
Full:
675409 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.7M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 82.1M 578M 81.5M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 82.3M 610M 11% 1.00x ONLINE -
=============
Results look weird, but if I do it manually, I can see that the space is cleared a while after the "rm" command is done.
[root at bczfsvm1test /test]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 620M 40.4M 619M /test
[root at bczfsvm1test /test]# zpool list test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 620M 72.2M 89% 1.00x ONLINE -
[root at bczfsvm1test /test]# rm -rf *
[root at bczfsvm1test /test]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 464M 196M 464M /test
[root at bczfsvm1test /test]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 464M 196M 464M /test
[root at bczfsvm1test /test]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 664K 659M 31K /test
[root at bczfsvm1test /test]# zfs list test
NAME USED AVAIL REFER MOUNTPOINT
test 664K 659M 31K /test
run with version 3 (b from 1..500, and sleep 30 added):
Run #2 (starts on 2 because accidently "c" was inserted before "run=1")
Full:
675394 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 702K 659M 31K /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 788K 691M 0% 1.00x ONLINE -
=============
Run #3
Full:
675405 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 710K 659M 31K /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 796K 691M 0% 1.00x ONLINE -
=============
Run #4
Full:
675395 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 708K 659M 31K /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 793K 691M 0% 1.00x ONLINE -
=============
Run #5
Full:
675410 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 696K 659M 31K /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 798K 691M 0% 1.00x ONLINE -
=============
Run #6
Full:
675277 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.9M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 648K 659M 31K /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 715K 691M 0% 1.00x ONLINE -
=============
Run #7
Full:
675391 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 706K 659M 31K /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 794K 691M 0% 1.00x ONLINE -
=============
Run #8
Full:
675267 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.9M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 714K 659M 31K /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 798K 691M 0% 1.00x ONLINE -
=============
Run #9
Full:
675392 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 700K 659M 31K /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 786K 691M 0% 1.00x ONLINE -
=============
Run #10
Full:
675394 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 422K 660M 31K /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 458K 692M 0% 1.00x ONLINE -
=============
Run #11
Full:
675398 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 406K 660M 31K /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 442K 692M 0% 1.00x ONLINE -
=============
Run #12
Full:
675418 /test
NAME USED AVAIL REFER MOUNTPOINT
test 660M 0 659M /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 660M 31.8M 95% 1.00x ONLINE -
Clear:
NAME USED AVAIL REFER MOUNTPOINT
test 436K 660M 31K /test
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
test 692M 472K 692M 0% 1.00x ONLINE -
=============
==============================
Conclusion
==============================
It seems that the maximum space varies by a small amount, and so does the "USED" and "ALLOC" when clearing the pool, but the free space is not clearly going down with each iteration of removing and filling.
If anything is missing it is probably less than a kilobyte after filling and removing all 11 times. (note that pool was never destroyed, and lots of test runs took place that are not recorded here, so there are many more than 11 runs, probably 40-50)
A repeat test on a larger disk, or with FreeBSD 9.0 may be appropriate.
More information about the freebsd-fs
mailing list