How to create holes in files ?

[Karli Sjöberg]

On tor, 2017-09-28 at 22:16 +0200, Ben RUBSON wrote:
> > 
> > On 28 Sep 2017, at 20:48, Karli Sjöberg <karli at inparadise.se>
> > wrote:
> > 
> > Den 28 sep. 2017 6:47 em skrev Ben RUBSON <ben.rubson at gmail.com>:
> > > 
> > > On 28 Sep 2017, at 18:34, Bob Eager wrote:
> > > 
> > > On Thu, 28 Sep 2017 17:26:09 +0200
> > > Fabian Keil wrote:
> > > 
> > > > 
> > > > Ben RUBSON wrote:
> > > > 
> > > > > 
> > > > > I'm trying to make holes in files in C.
> > > > > Goal is to deallocate huge files on ZFS while (randomly)
> > > > > reading
> > > > > them. 
> > > > My interpretation of the above is that you want to create holes
> > > > without changing the file size and without affecting data that
> > > > is located before or after the holes that you want to create.
> > > > 
> > > > Otherwise you could simply "deallocate" the content with
> > > > truncate(1).
> > > If he doesn't mind copying the files, dd(1) will do the job.
> > > However, I
> > > expect that doesn't meet his criteria.
> > Thank you Bob for your suggestion.
> > You're right goal is to avoid copying data : free space would not
> > necessarily
> > allow this, and as the files I'm working on are some hundreds of GB
> > in size,
> > it would really be a counterproductive long stressing storage
> > operation.
> > 
> > Well, correct me if I'm wrong, but wouldn't dd with "seek" mitigate
> > the issue of writing out all of the data from beginning to end. If
> > you seek from the beginning of the file to the point you want to
> > start writing from and use bs to specify how large of a "hole" you
> > want, the operation wouldn't take long at all. You would, in my
> > opinion, achieve exactly what you want, to "create holes in files".
> > Am I wrong?
> I would have liked to do this in C.

Yepp, I understand that, I was just bringing up the idea. But hey, if
dd can do it, just look in the source of how it does it?

> And the holes I need to create are not necessarily at the beginning
> of a file.

No, you _do not_ have to create holes in the beginning of a file, you
can do it anywhere in the file you like. E.g. it´s the trick I use to
"clean" hardrives to make them look empty, by only erasing the first
and last 10 MB of the drives, a whole JBOD only takes seconds to clean.
Say you have a 10 MB large file and you want to make a 1 MB large hole
somewhere in the middle of it, you do it like this:

# dd if=/dev/zero of=/foo/bar.bin bs=1M seek=7

It´ll make the file look like this inside (hope the ASCII gods are with
me):
 ____________
|_______|_|__|

Important to know is that "bs" determines the unit by wich you "seek"!

I don´t know if this might be a solution to your problem, just throwing
it out there.

/K

> I randomly read parts of huge input files that I need to "free" on
> the fly
> to recover storage space for the output files.
> 
> Ben
> 
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"