bad news for bsdtar..

[Tim Kientzle]

Julian Elischer wrote:
> done on a small subdirectory of our data...
> (only accounts starting with 849xxx)
> total data as reported by 'du': 9394486 KB  (9GB)
> (note du takes links into account and doesn't count the same data twice)
> number of files with > 1 link: 
> # find 849* -type f -links +1 | wc -l
>   240395
> #
> total number of file directory entries..
> # find 849* -type f | wc -l
>   598018
> #
> total number of directories:
> %find 849* -type d | wc -l
>    51245
> 
> average filesize 23k
> but major file types are: jpeg images.. 100kb
> 			jpeg thumbnails, 3kb
> 			transaction descriptors 1kb
> ########################################################
> the bad news:
> 
> old (FreeBSD 2.x) tar with symlink hack
> %time ~/tar cf /dev/null 849*
> 5.309u 73.584s 21:15.44 6.1%    329+5150k 786697+0io 0pf+0w
> 
> freebsd 4.8 tar
> %time tar cf /dev/null 849
> 5.159u 17.724s 4:08.97 9.1%     364+5863k 66144+0io 1pf+0w
> 
> bsdtar
> %time ~/bsdtar cf /dev/null 849*
> 16.423u 68.141s 31:40.82 4.4%   402+1639k 784651+0io 0pf+0w
> 
> It looks to me that the tar in 4.8 is doing something very
> sneeky with it's IOs..
> maybe using mmap?
> maybe detecting /dev/null and instead discarding the data?
> maybe reading the entire file in a single read?

Very interesting.  bsdtar, I note, used about 1/4 as
much memory as either of the others.

I'd be interested to know what happens if you edit
write.c to change the buffer used in write_file_data
from the current miserly 8k to something more reasonable,
like 128k.

I love a challenge... ;-)

Tim