How to create holes in files ?

[Ben RUBSON]

> On 29 Sep 2017, at 07:54, Karli Sjöberg <karli at inparadise.se> wrote:
> 
> On tor, 2017-09-28 at 22:16 +0200, Ben RUBSON wrote:
>>> 
>>> On 28 Sep 2017, at 20:48, Karli Sjöberg <karli at inparadise.se>
>>> wrote:
>>> 
>>> Den 28 sep. 2017 6:47 em skrev Ben RUBSON <ben.rubson at gmail.com>:
>>>> 
>>>> On 28 Sep 2017, at 18:34, Bob Eager wrote:
>>>> 
>>>> On Thu, 28 Sep 2017 17:26:09 +0200
>>>> Fabian Keil wrote:
>>>> 
>>>>> 
>>>>> Ben RUBSON wrote:
>>>>> 
>>>>>> 
>>>>>> I'm trying to make holes in files in C.
>>>>>> Goal is to deallocate huge files on ZFS while (randomly)
>>>>>> reading
>>>>>> them. 
>>>>> My interpretation of the above is that you want to create holes
>>>>> without changing the file size and without affecting data that
>>>>> is located before or after the holes that you want to create.
>>>>> 
>>>>> Otherwise you could simply "deallocate" the content with
>>>>> truncate(1).
>>>> If he doesn't mind copying the files, dd(1) will do the job.
>>>> However, I
>>>> expect that doesn't meet his criteria.
>>> Thank you Bob for your suggestion.
>>> You're right goal is to avoid copying data : free space would not
>>> necessarily
>>> allow this, and as the files I'm working on are some hundreds of GB
>>> in size,
>>> it would really be a counterproductive long stressing storage
>>> operation.
>>> 
>>> Well, correct me if I'm wrong, but wouldn't dd with "seek" mitigate
>>> the issue of writing out all of the data from beginning to end. If
>>> you seek from the beginning of the file to the point you want to
>>> start writing from and use bs to specify how large of a "hole" you
>>> want, the operation wouldn't take long at all. You would, in my
>>> opinion, achieve exactly what you want, to "create holes in files".
>>> Am I wrong?
>> I would have liked to do this in C.
> 
> Yepp, I understand that, I was just bringing up the idea. But hey, if
> dd can do it, just look in the source of how it does it?

Of course yes, you're right :)

>> And the holes I need to create are not necessarily at the beginning
>> of a file.
> 
> No, you _do not_ have to create holes in the beginning of a file, you
> can do it anywhere in the file you like. E.g. it´s the trick I use to
> "clean" hardrives to make them look empty, by only erasing the first
> and last 10 MB of the drives, a whole JBOD only takes seconds to clean.
> Say you have a 10 MB large file and you want to make a 1 MB large hole
> somewhere in the middle of it, you do it like this:
> 
> # dd if=/dev/zero of=/foo/bar.bin bs=1M seek=7
> 
> It´ll make the file look like this inside (hope the ASCII gods are with
> me):
>  ____________
> |_______|_|__|

Unfortunately here storage blocks will be set/written to 0,
they will not be freed/recovered, a file hole will not be created.
Below are some commands which demonstrate this.
The last 3/4 commands show a _newly_ created file with a hole.

# zfs userspace home
POSIX User  root  36.5K   none

# dd if=/dev/random of=f bs=1M count=4
4+0 records out

# zfs userspace home
POSIX User  root  4.04M   none

# dd if=/dev/zero of=f bs=1M count=2 seek=2
2+0 records out

# zfs userspace home
POSIX User  root  4.04M   none

# rm f

# dd if=/dev/zero of=f bs=1M count=2 seek=2
2+0 records out

# zfs userspace home
POSIX User  root  2.04M   none

# ls -lh f
-rw-------  1 root  wheel   4.0M 29 Sep 08:19 f