Need a filesystem with "unlimited" inodes
Karl Vogel
vogelke+unix at pobox.com
Wed Jun 10 19:06:32 UTC 2009
>> On Tue, 9 Jun 2009 03:10:46 am Matthew Seaman wrote:
M> Or store your data in a RDBMS rather than in the filesystem.
>> On Tue, 9 Jun 2009 09:45:48 -0500, Kirk Strauser <kirk at strauser.com> said:
K> Hear, hear. I'm hard pressed to imagine why you'd need 100M 1KB files.
DBs are great when you have structured data, but semi-structured text
(like email) makes for a very poor fit. To see why, have a look at
http://www.memoryhole.net/~kyle/databaseemail.html
If you really need to store 100 million smallish chunks of information,
consider using zip. Create 256 folders named 00-ff:
#!/bin/sh
hex='0 1 2 3 4 5 6 7 8 9 a b c d e f'
for x in $hex ; do
for y in $hex ; do
mkdir ${x}${y}
done
done
exit 0
Use the hash of your choice to map the name of each chunk to one of 256
zipfiles under each directory. This gives you 64k zipfiles, and if you
put 1500 or so chunks in each one, you're pretty close to 100 million.
me% cat mkchunks
#!/usr/bin/perl -w
for $chunk (@ARGV) {
$_ = chunk2file($chunk);
$file = "$1/$2.zip" if m/(..)(..)/;
print "$file $chunk\n";
}
exit(0);
sub chunk2file {
my $str = shift;
my ($byte, $sum);
use integer;
$sum = 0;
foreach $byte (unpack("C*", $str)) { # SDBM hash
$sum = $byte + 65587 * $sum;
}
$sum &= 0xffff; # keep lowest 16 bits
no integer;
return sprintf("%4.4x", $sum);
}
me% ./mkchunks freebsd solaris
16/f7.zip freebsd
ca/1f.zip solaris
You'll get a better distribution if you use a hash like Digest::SHA1.
--
Karl Vogel I don't speak for the USAF or my company
People like you are the reason people like me need medication. --bumper sticker
More information about the freebsd-questions
mailing list