effect of strip(1) on du(1)
Allan Jude
allanjude at FreeBSD.org
Fri Mar 3 17:25:09 UTC 2017
On March 3, 2017 9:11:30 AM EST, "Rodney W. Grimes" <freebsd-rwg at pdx.rh.CN85.dnsmgr.net> wrote:
>-- Start of PGP signed section.
>[ Charset ISO-8859-1 unsupported, converting... ]
>> On 2017-Mar-02 22:19:10 -0800, "Rodney W. Grimes"
><freebsd-rwg at pdx.rh.CN85.dnsmgr.net> wrote:
>> >> du(1) is using fts_read(3), which is based on the stat(2)
>information.
>> >> The OpenGroup defines st_blocksize as "Number of blocks allocated
>for
>> >> this object." In the case of ZFS, a write(2) may return before
>any
>> >> blocks are actually allocated. And thanks to compression, gang
>> ...
>> >My gut tells me that this is gona cause problems, is it ONLY
>> >the st_blocksize data that is incorrect then not such a big
>> >problem, or are we returning other meta data that is wrong?
>>
>> Note that it's st_blocks, not st_blocksize.
>Yes, I just ignore that digretion, as well as the digretion into
>fts_read
>being anything special about this, as it just ends up calling stat(2)
>in
>the end anyway.
>
>>
>> I did an experiment, writing a (roughly) 113MB file (some data I had
>> lying around), close()ing it and then stat()ing it in a loop. This
>is
>> FreeBSD 10.3 with ZFS and lz4 compression. Over the 26ms following
>the
>> close(), st_blocks gradually rose from 24169 to 51231. It then
>stayed
>> stable until 4.968s after the close, when st_blocks again started
>> increasing until it stabilized after a total of 5.031s at 87483.
>Based
>> on this, st_blocks reflects the actual number of blocks physically
>> written to disk. None of the other fields in the struct stat vary.
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>Thank you for doing the proper regression test, that satisfies me that
>we dont have a lattent bug sitting here and infact what we have is
>exposure of the kernel caching, which I might be too thrilled about,
>is just how its gona have to be.
>
>>
>> The 5s delay is presumably the TXG delay (since this system is
>basically
>> unloaded). I'm not sure why it writes roughly ? the data immediately
>> and the rest as part of the next TXG write.
>>
>> >My expectactions of executing a stat(2) call on a file would
>> >be that the data returned is valid and stable. I think almost
>> >any program would expect that.
>>
>> I think a case could be made that st_blocks is a valid representation
>> of "the number of blocks allocated for this object" - with the number
>> increasing as the data is physically written to disk. As for it
>being
>> stable, consider a (hypothetical) filesystem that can transparently
>> migrate data between different storage media, with different
>compression
>> algorithms etc (ZFS will be able to do this once the mythical block
>> rewrite code is written).
>
>I could counter argue that st_blocks is:
>st_blocks The actual number of blocks allocated for the file in
> 512-byte units.
>
>Nothing in that says anything about "on disk". So while this thing
>is sitting in memory on the TXG queue we should return the number of
>512 byte blocks used by the memory holding the data.
>I think that would be the more correct thing than exposing the
>fact this thing is setting in a write back cache to userland.
Can we compare the results of du with du -A?
Du will show compression savings, and -A wont
ZFS compresses between the write cache and the disk, so the final size may not be know for 5+ seconds
--
Allan Jude
More information about the freebsd-hackers
mailing list