[Bug 299361] /bin/stat -h output is not parsable, is not scalable, lacks locking, has no option for listing data sections
Date: Thu, 15 Oct 2026 00:21:33 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=299361
Enji Cooper <ngie@FreeBSD.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |ngie@FreeBSD.org
--- Comment #1 from Enji Cooper <ngie@FreeBSD.org> ---
In summary, in order to represent the data in a scalable way, I would strongly
encourage writing another tool which could massage the data into a form that
you can consume for the time being, e.g., using C, C++, golang, python (less
encouraged because of process overhead, but you could go that route), etc.
> - stat -h produces an endless stream of values in a single line, but standard /bin/sh and other tools have line length limitations. Writing scripts which can actually parse this are HARD to write, or trigger hidden limitations in tools once the number of holes in a sparse file gets longer
/bin/sh is the wrong tool for the job IMHO. C/C++, golang, or python seems more
apt for items of that scale.
If you read the output using read(2), you could get by with a more scalable
wrapper, but I would strongly discourage that.
> - stat -h always starts at file position zero, which renders it unusable in real world multi-PB sparse files. There should be an option to pass startpos,endpos to limit the search, allowing scripts to work on only parts of the file
You could do that using dd today with the bs, count, and iseek arguments. Not
the ideal solution for you probably, but it's one that exists today. I would
not recommend using dd though because then you would have to arthimetic to
figure out where the initial position is, then go from there.
Again, C/C++, golang, or python seems more apt for this.
> - stat -h lacks an option to lock the file to prevent modifications by other tools while it is looking for holes
This part at least is documented in the manpage:
> Please note that the only way to retrieve information about the holes in a file is to open it and walk the list of holes and data regions using lseek(2). If the file is being modified by another process at the same time as stat is
> inspecting it, the result may be inconsistent.
It's easy to solve using flock(2) or open(2) in an application. Not sure if
it's wise for stat(1) to grow that support.
The catch in terms of implementing the locking in stat(1) is doing this could
cause application instability (hanging processes, etc), which could result in
support headaches if someone relies on the command to do what you are asking
for. It would also require the applications themselves to actually open the
files using `O_SHLOCK` or `O_EXLOCK`.
> - stat -h always starts at file position zero, which renders it unusable in real world multi-PB sparse files. There should be an option to pass startpos,endpos to limit the search, allowing scripts to work on only parts of the file
I want to focus on this part first:
> real world multi-PB sparse files
Is that really an average real-world scenario given storage systems for the
average user? This seems like it's a valid edgecase for more scalable storage
solutions, like NetApp or OneFS scale solutions, but it seems a highly unlikely
in the average user scenario. These products should have tools which provide
this information for you.
Apart from that, you need to roll your own tool (see my comments below).
> - stat -h lacks an option to list the positions of data sections
Discerning the data sections from the holes is a fair critique.
From stat(1):
> Each hole is reported as its starting offset as a decimal number followed
> by a hyphen and the ending offset (one less than the starting
> offset of the data region that follows the hole) as a decimal
> number. If the file ends in a hole, the ending offset of the
> final hole will be one less than the size of the file.
```
% uname -or; sysctl kern.osreldate
FreeBSD 16.0-CURRENT
kern.osreldate: 1600027
% truncate -s 1g /tmp/bogus
% dd if=/dev/zero of=/tmp/bogus oseek=1g count=5
5+0 records in
5+0 records out
2560 bytes transferred in 0.000237 secs (10795493 bytes/sec)
% stat -h /tmp/bogus
0-1073709055,1073741824-549755781119,549755816448 /tmp/bogus
```
What I did was truncate the file at 1g and appended 512 * 5 = 2560 zeros to the
end of the file.
The first region is a hole and the second region is a data region, and I assume
(but am not 100% sure) that the last item in the list corresponds with the real
world block/fragment representation on disk.
Here's the code that does this in stat(1) [`fdlistholes(..)` in
`usr.bin/stat/stat.c`]:
```
for (;;) {
if ((off = lseek(fd, pos, SEEK_HOLE)) < 0) {
if (errno != ENXIO)
return (-1);
/*
* This can only happen if the file was
* truncated while we were scanning it, or
* on the initial seek if the file is
* empty. Report the virtual hole at the
* end of the file at this position.
*/
off = pos;
}
printf("%jd", (intmax_t)off);
pos = off;
if ((off = lseek(fd, pos, SEEK_DATA)) < 0) {
if (errno != ENXIO)
return (-1);
/*
* There are no more data regions in the
* file, or it got truncated. However, we
* may not be at the end yet.
*/
if ((off = lseek(fd, 0, SEEK_END)) > pos)
printf("-%jd", (intmax_t)off - 1);
break;
}
printf("-%jd,", (intmax_t)off - 1);
pos = off;
}
```
What you would need to do is open the file with `O_RDONLY|O_EXLOCK` beforehand,
like so:
```
int fd = open(<path>, O_RDONLY|O_EXLOCK|O_NONBLOCK);
if (fd == -1)
err(1, "open failed");
```
Then iterate through, distinguishing the SEEK_DATA and SEEK_HOLE output using a
meaningful piece of metadata, e.g., a prefix or suffix.
--
You are receiving this mail because:
You are the assignee for the bug.