How to create holes in files ?

[Ben RUBSON]

> On 29 Sep 2017, at 08:33, Karli Sjöberg <karli at inparadise.se> wrote:
> 
> On fre, 2017-09-29 at 08:26 +0200, Ben RUBSON wrote:
>>> 
>>> On 29 Sep 2017, at 07:54, Karli Sjöberg <karli at inparadise.se>
>>> wrote:
>>> 
>>> On tor, 2017-09-28 at 22:16 +0200, Ben RUBSON wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> On 28 Sep 2017, at 20:48, Karli Sjöberg <karli at inparadise.se>
>>>>> wrote:
>>>>> 
>>>>> Den 28 sep. 2017 6:47 em skrev Ben RUBSON <ben.rubson at gmail.com
>>>>>> :
>>>>>> 
>>>>>> 
>>>>>> On 28 Sep 2017, at 18:34, Bob Eager wrote:
>>>>>> 
>>>>>> On Thu, 28 Sep 2017 17:26:09 +0200
>>>>>> Fabian Keil wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Ben RUBSON wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I'm trying to make holes in files in C.
>>>>>>>> Goal is to deallocate huge files on ZFS while (randomly)
>>>>>>>> reading
>>>>>>>> them. 
>>>>>>> My interpretation of the above is that you want to create
>>>>>>> holes
>>>>>>> without changing the file size and without affecting data
>>>>>>> that
>>>>>>> is located before or after the holes that you want to
>>>>>>> create.
>>>>>>> 
>>>>>>> Otherwise you could simply "deallocate" the content with
>>>>>>> truncate(1).
>>>>>> If he doesn't mind copying the files, dd(1) will do the job.
>>>>>> However, I
>>>>>> expect that doesn't meet his criteria.
>>>>> Thank you Bob for your suggestion.
>>>>> You're right goal is to avoid copying data : free space would
>>>>> not
>>>>> necessarily
>>>>> allow this, and as the files I'm working on are some hundreds
>>>>> of GB
>>>>> in size,
>>>>> it would really be a counterproductive long stressing storage
>>>>> operation.
>>>>> 
>>>>> Well, correct me if I'm wrong, but wouldn't dd with "seek"
>>>>> mitigate
>>>>> the issue of writing out all of the data from beginning to end.
>>>>> If
>>>>> you seek from the beginning of the file to the point you want
>>>>> to
>>>>> start writing from and use bs to specify how large of a "hole"
>>>>> you
>>>>> want, the operation wouldn't take long at all. You would, in my
>>>>> opinion, achieve exactly what you want, to "create holes in
>>>>> files".
>>>>> Am I wrong?
>>>> I would have liked to do this in C.
>>> Yepp, I understand that, I was just bringing up the idea. But hey,
>>> if
>>> dd can do it, just look in the source of how it does it?
>> Of course yes, you're right :)
>> 
>>> 
>>>> 
>>>> And the holes I need to create are not necessarily at the
>>>> beginning
>>>> of a file.
>>> No, you _do not_ have to create holes in the beginning of a file,
>>> you
>>> can do it anywhere in the file you like. E.g. it´s the trick I use
>>> to
>>> "clean" hardrives to make them look empty, by only erasing the
>>> first
>>> and last 10 MB of the drives, a whole JBOD only takes seconds to
>>> clean.
>>> Say you have a 10 MB large file and you want to make a 1 MB large
>>> hole
>>> somewhere in the middle of it, you do it like this:
>>> 
>>> # dd if=/dev/zero of=/foo/bar.bin bs=1M seek=7
>>> 
>>> It´ll make the file look like this inside (hope the ASCII gods are
>>> with
>>> me):
>>>  ____________
>>>> 
>>>> _______|_|__|
>> Unfortunately here storage blocks will be set/written to 0,
>> they will not be freed/recovered, a file hole will not be created.
>> Below are some commands which demonstrate this.
>> The last 3/4 commands show a _newly_ created file with a hole.
>> 
>> # zfs userspace home
>> POSIX User  root  36.5K   none
>> 
>> # dd if=/dev/random of=f bs=1M count=4
>> 4+0 records out
>> 
>> # zfs userspace home
>> POSIX User  root  4.04M   none
>> 
>> # dd if=/dev/zero of=f bs=1M count=2 seek=2
>> 2+0 records out
>> 
>> # zfs userspace home
>> POSIX User  root  4.04M   none
>> 
>> # rm f
>> 
>> # dd if=/dev/zero of=f bs=1M count=2 seek=2
>> 2+0 records out
>> 
>> # zfs userspace home
>> POSIX User  root  2.04M   none
>> 
>> # ls -lh f
>> -rw-------  1 root  wheel   4.0M 29 Sep 08:19 f
> 
> Ah, I see, nice demonstration! But wouldn´t TRIM take of that though
> (if that´s available)?

Good question ! I'm however not sure TRIM would notify ZFS about
space not consumed. In addition I'm mainly on non-SSD drives :-/

Ben