How to create holes in files ?

[Karli Sjöberg]

On fre, 2017-09-29 at 08:54 +0200, Ben RUBSON wrote:
> > 
> > On 29 Sep 2017, at 08:33, Karli Sjöberg <karli at inparadise.se>
> > wrote:
> > 
> > On fre, 2017-09-29 at 08:26 +0200, Ben RUBSON wrote:
> > > 
> > > > 
> > > > 
> > > > On 29 Sep 2017, at 07:54, Karli Sjöberg <karli at inparadise.se>
> > > > wrote:
> > > > 
> > > > On tor, 2017-09-28 at 22:16 +0200, Ben RUBSON wrote:
> > > > > 
> > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > On 28 Sep 2017, at 20:48, Karli Sjöberg <karli at inparadise.s
> > > > > > e>
> > > > > > wrote:
> > > > > > 
> > > > > > Den 28 sep. 2017 6:47 em skrev Ben RUBSON <ben.rubson at gmail
> > > > > > .com
> > > > > > > 
> > > > > > > :
> > > > > > > 
> > > > > > > 
> > > > > > > On 28 Sep 2017, at 18:34, Bob Eager wrote:
> > > > > > > 
> > > > > > > On Thu, 28 Sep 2017 17:26:09 +0200
> > > > > > > Fabian Keil wrote:
> > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > Ben RUBSON wrote:
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > I'm trying to make holes in files in C.
> > > > > > > > > Goal is to deallocate huge files on ZFS while
> > > > > > > > > (randomly)
> > > > > > > > > reading
> > > > > > > > > them. 
> > > > > > > > My interpretation of the above is that you want to
> > > > > > > > create
> > > > > > > > holes
> > > > > > > > without changing the file size and without affecting
> > > > > > > > data
> > > > > > > > that
> > > > > > > > is located before or after the holes that you want to
> > > > > > > > create.
> > > > > > > > 
> > > > > > > > Otherwise you could simply "deallocate" the content
> > > > > > > > with
> > > > > > > > truncate(1).
> > > > > > > If he doesn't mind copying the files, dd(1) will do the
> > > > > > > job.
> > > > > > > However, I
> > > > > > > expect that doesn't meet his criteria.
> > > > > > Thank you Bob for your suggestion.
> > > > > > You're right goal is to avoid copying data : free space
> > > > > > would
> > > > > > not
> > > > > > necessarily
> > > > > > allow this, and as the files I'm working on are some
> > > > > > hundreds
> > > > > > of GB
> > > > > > in size,
> > > > > > it would really be a counterproductive long stressing
> > > > > > storage
> > > > > > operation.
> > > > > > 
> > > > > > Well, correct me if I'm wrong, but wouldn't dd with "seek"
> > > > > > mitigate
> > > > > > the issue of writing out all of the data from beginning to
> > > > > > end.
> > > > > > If
> > > > > > you seek from the beginning of the file to the point you
> > > > > > want
> > > > > > to
> > > > > > start writing from and use bs to specify how large of a
> > > > > > "hole"
> > > > > > you
> > > > > > want, the operation wouldn't take long at all. You would,
> > > > > > in my
> > > > > > opinion, achieve exactly what you want, to "create holes in
> > > > > > files".
> > > > > > Am I wrong?
> > > > > I would have liked to do this in C.
> > > > Yepp, I understand that, I was just bringing up the idea. But
> > > > hey,
> > > > if
> > > > dd can do it, just look in the source of how it does it?
> > > Of course yes, you're right :)
> > > 
> > > > 
> > > > 
> > > > > 
> > > > > 
> > > > > And the holes I need to create are not necessarily at the
> > > > > beginning
> > > > > of a file.
> > > > No, you _do not_ have to create holes in the beginning of a
> > > > file,
> > > > you
> > > > can do it anywhere in the file you like. E.g. it´s the trick I
> > > > use
> > > > to
> > > > "clean" hardrives to make them look empty, by only erasing the
> > > > first
> > > > and last 10 MB of the drives, a whole JBOD only takes seconds
> > > > to
> > > > clean.
> > > > Say you have a 10 MB large file and you want to make a 1 MB
> > > > large
> > > > hole
> > > > somewhere in the middle of it, you do it like this:
> > > > 
> > > > # dd if=/dev/zero of=/foo/bar.bin bs=1M seek=7
> > > > 
> > > > It´ll make the file look like this inside (hope the ASCII gods
> > > > are
> > > > with
> > > > me):
> > > >  ____________
> > > > > 
> > > > > 
> > > > > _______|_|__|
> > > Unfortunately here storage blocks will be set/written to 0,
> > > they will not be freed/recovered, a file hole will not be
> > > created.
> > > Below are some commands which demonstrate this.
> > > The last 3/4 commands show a _newly_ created file with a hole.
> > > 
> > > # zfs userspace home
> > > POSIX User  root  36.5K   none
> > > 
> > > # dd if=/dev/random of=f bs=1M count=4
> > > 4+0 records out
> > > 
> > > # zfs userspace home
> > > POSIX User  root  4.04M   none
> > > 
> > > # dd if=/dev/zero of=f bs=1M count=2 seek=2
> > > 2+0 records out
> > > 
> > > # zfs userspace home
> > > POSIX User  root  4.04M   none
> > > 
> > > # rm f
> > > 
> > > # dd if=/dev/zero of=f bs=1M count=2 seek=2
> > > 2+0 records out
> > > 
> > > # zfs userspace home
> > > POSIX User  root  2.04M   none
> > > 
> > > # ls -lh f
> > > -rw-------  1 root  wheel   4.0M 29 Sep 08:19 f
> > Ah, I see, nice demonstration! But wouldn´t TRIM take of that
> > though
> > (if that´s available)?
> Good question ! I'm however not sure TRIM would notify ZFS about
> space not consumed. In addition I'm mainly on non-SSD drives :-/

OK, yeah it was more of a general question, not concerning your
specific setup. Maybe someone else has an idea? Anyhow, I hope you get
a better answer to your specific question.

/K

> 
> Ben
> 
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"