scripting the buildworld/installworld process

[Kent Stewart]

On Sunday 31 August 2003 04:05 pm, Charles Howse wrote:
> > bw w -j5
> > 893.563u 311.353s 30:31.44 65.7%        1302+1591k 53927+138930io
> > 2331pf+0w
> >
> > The wall clock time is really how efficient your process is. The
> > fact that your user time or sys time is faster doesn't mean much if
> > the wall
> > clock time is 1/3 longer.
>
> Wall clock?  Would that be 30:31.44 from above?
>
> > I think that once you have it setup, I will probably make my
> > KISS simple
> > script more complicated :). I have one machine that the mouse
> > doesn't
>
> Glad to share, if that's what you mean.

That is what I meant. You learn by trying things.

>
> After re-reading your page about scripting the buildworld, I see an
> opportunity to test my script.
>
> I never thought about putting /usr/src and /usr/obj on different
> HDD's. I have 2 ATAPI HDD in my box, on different IDE busses, so that
> should work out well.
> I'll also need to adjust some slice sizes as well.
>
> How much space do you recommend for /var if I'm logging like you do?
> It's currently:
> Disk1 -
> / 100M
> swap 256M
> /var 200M
> /usr <remainder> about 1.8G
> Disk2 -
> /disk2 <entire disk> 7.8G

My smallest HD is 13GB and the system had 4 of them. My typical system 
is on a  3x30GB ATA-100/133 HDs. Since, I have a lot of free space, my 
partitions are typically 1.5GB. FreeBSD 4.x creates an ../obj of around 
300-400MB; however, FreeBSD 5.x produces a 700-800MB. 

I like to mount a partition as /usr/obj and /usr/src. The builds work 
just fine with a link to /disk2/obj as /usr/obj. The reason for 
separate partition is that I have messed up my system to the point that 
I had to nfs_mount /usr/src and /usr/obj from a different machine. If I 
mount /disk2/obj as /usr/obj, the make installworld dies because it 
knows the true path to the obj files. 

If I redo a system, I create a mount point as /usr/obj. Until then, I 
just link /usr/obj to /usr2/obj and etc. I think that just moving 
/usr/obj to your 2nd HD as a link would separate the I/O.

For FreeBSD 5.x, you need a lot more space in /. When I installed 
5.0-release, I created / as 500MB and it is about 24% used. If you have 
to manually download some tarballs, you can chew up space in a hurry.

The files in /var/log/build can fill up a fs really fast because each 
system update is around 7-8MB. Since I log everything from cvsup to 
"make index", I fill up space in a hurry :). If something breaks, I 
want to be able to point to the commit that broke it.

>
> So I assume /usr/obj should be on the second HDD?
> About how much space should I reserve?
>
> Looks like I should also adjust my script to remove the -j4, eh?

Like you said, play with it. I use write caching and softupdates. They 
all made it faster. Depending on your system, you may find that -j is 
faster. From my experience, all of my systems (6 of various speeds and 
cpus), do a build world faster without a -j.

When I was doing benchmarks on a Cray X/MP with 100MB/s & 1000MB/s data 
channels to stripped disks capable of 20MB/s continuous, write caching 
doubled the throughput of the system. The gain was not obvious up front 
and everyone was surprised by the result. After we saw it, it was a 
shoulder shrug because it appeared obvious that you want a process to 
have faster access to data that it needs than forcing it to save 
processed data first. In the commercial world, protecting the processed 
data has a much higher assigned value. I went for speed. If someone 
said 4-stable was broken, I wanted to be able to test the builds in 
less than 30 minutes. I also didn't want to spend a lot of money to be 
able to do that :).

BTW, those build times all had setiathome running in the background. 
Having it running in the background really doesn't change the elapsed 
time too much exept in 5.x. I kill seti before I do a timed run on it.

Kent

-- 
Kent Stewart
Richland, WA

http://users.owt.com/kstewart/index.html