Performance Tracker project update
erik at cederstrand.dk
Fri Feb 8 09:00:06 UTC 2008
Brooks Davis skrev:
> On Wed, Jan 30, 2008 at 07:20:23PM +0100, Erik Cederstrand wrote:
>> I'd like a situation where I can very quickly set up a slave with a
>> specific version of FreeBSD to run additional tests or provide shell access
>> to a developer. This currently involves adding an entry to a queue,
>> rebooting and waiting 2 minutes. Quick and easy, but the archiving strategy
>> is obviously very inefficient.
>> I'm thinking of a couple of options:
>> 1. Having one full install per month and archiving the rest as diffs
>> against that by recursively bsdiff'ing every file in the tree (I
>> could bsdiff a whole tarball, but bsdiff is very memory-intensive).
>> Quick test: 25 mins.
>> 2. Make a hash of all files and only store the binaries where the hash
>> is different from the monthly tarball. Faster than 1., but less
>> effective. Quick test: 5 mins.
>> 3. Use some kind of VCS. My experience with Subversion and binary files
>> is that it's very slow.
>> 4. Throw hardware at the problem.
>> I'd say it should not take more than 10 mins to recreate an archived
>> version. Any thoughts?
> It seems like you should be able to combine 1 and 2 with checksums to
> decide if you need to run diffs. I'd think that would be quite fast.
I finally got around to testing this, and with a combination of mtree
comparing md5 hashes, bsdiff compacting changed files and hardlinking
unchanged files I get a reduction in size from 256MB to 10MB. Pretty
good, and the whole operation only takes a few minutes.
I have one peculiarity, though. I install python2.5 into the directory
containing the build, and even though the python version has not
changed, I still get mismatching md5 sums on every .pyo and .pyc file.
Any thoughts on this?
More information about the freebsd-performance