scripting the buildworld/installworld process

Sun Aug 31 15:00:55 PDT 2003

On Sunday 31 August 2003 02:19 pm, Charles Howse wrote:
> > > > > I'll be glad to post the scripts for review if anyone's
> > > > > interested.
> > > >
> > > > I'll take a look and test them for you also.
> > >
> > > OK, any input or corrections welcome.
> > > I definitly would be interested in more error checking and
> > > reporting, and I'm thinking about using \time to append the
> > > time it takes the long utilities to run to a file for comparison
> > > purposes.
> > >
> > > ------ This is the first one ------
> > > #!/usr/local/bin/bash
> > > #
> > > # Update the system, run in multi-user mode
> > >
> > > echo -n "Have you read /usr/src/UPDATING? [y/n]: "
> > > read a
> > > if [ $a = n ] ; then
> > >     less /usr/src/UPDATING
> > >     exit
> > > fi
> > > echo -n "Have you merged /etc/group and /usr/src/etc/group?
> > > [y/n]: " read b
> > > if [ $b = n ] ; then
> > >     diff -c /etc/group /usr/src/etc/group | less
> > >     exit
> > > fi
> > > echo -n "Have you merged /etc/master.passwd and
> > > /usr/src/etc/master.passwd? [y/n]: "
> > > read c
> > > if [ $c = n ] ; then
> > >     diff -c /etc/master.passwd /usr/src/etc/master.passwd
> > >     exit
> > > fi
> > > # copy my customized make.conf
> > > cp /disk2/larry/etc/make.conf /etc
> > > # copy my customized kernel config file
> > > cp /disk2/larry/usr/src/sys/i386/conf/CUSTOM
> > > /usr/src/sys/i386/conf echo -n "Clean out /usr/obj? [y/n]: "
> > > read d
> > > if [ $d = y ] ; then
> > >     cd /usr/obj
> > >     chflags -R noschg *
> > >     rm -rf *
> > > fi
> > > echo -n "Continue with build? [y/n]: "
> > > read e
> > > if [ $e = n ] ; then
> > >     exit
> > > fi
> > > cd /usr/src
> > > make -j4 buildworld
> > > make buildkernel KERNCONF=GENERIC
> > > make installkernel KERNCONF=GENERIC
> > > cp /boot/kernel /boot/kernel.GENERIC
> > > make buildkernel KERNCONF=CUSTOM
> > > make installkernel KERNCONF=CUSTOM
> > > echo "Reboot to single user mode and run
> > > /disk2/larry/bin/update2" Exit
> > >
> > > ------ This is the second one ------
> > > #!/usr/local/bin/bash
> > > #
> > > # Update the system, run in single user mode
> > >
> > > echo -n "Continue with installworld? [y/n]: "
> > > read a
> > > if [ $a = n ] ; then
> > >     exit
> > > fi
> > > make installworld
> > > if [ -d /etc.old ] ; then
> > >     rm -r /etc.old
> > > fi
> > > cp -Rp /etc /etc.old
> > > /usr/sbin/mergemaster -a
> > > cd /dev
> > > ./MAKEDEV all
> > > cd /usr/src/release/sysinstall
> > > make clean
> > > make all install
> > > echo "Finished!  Do ps and top work?"
> > > Exit
> > >
> > > > > 2. After reading man mergemaster, I see that the -a
> >
> > flag will run
> >
> > > > > mergemaster automatically and leave any new files in
> > > >
> > > > /var/tmp/temproot.
> > > >
> > > > > I'd like to confirm that if I opt to run mergemaster -a,
> > > >
> > > > then I should
> > > >
> > > > > check/merge any files before leaving single user mode.  It
> > > >
> > > > seems like
> > > >
> > > > > the right thing to do...?
> > > >
> > > > Correct, but I'll advise you run mergemaster manually.
> > > > Margemaster is such a dangerous thing to run with -a on a
> > > > system that has had so many mods to /etc
> > >
> > > When I upgraded from 4.8-p3 to p4 recently, I ran mergemaster
> > > manually. The only file I had to make a decision on was (I
> >
> > forget the
> >
> > > name) the one that prints the uname information when booting.
> > > Just for the sake of discussion, let's say I did opt to run
> > > mergemaster -a.
> > > All I would have to do is check every file in /var/tmp/temproot
> > > and be sure it didn't break anything by overwriting the older
> > > version, correct?
> >
> > One area I would check is /etc/master.passwd and groups. It would
> > be really interesting for you if your user accounts all
> > disappeared. I also won't let mergemaster touch my hosts and
> > printcap files.
>
> I'm not sure exactly what you mean.  I'm asking whether or not to
> diff the master.passwd and group files ( see above ).
> If I run mergemaster -a, my understanding is that it doesn't make any
> changes, it just leaves the new files in /var/tmp/temproot to be
> dealt with later, so it wouldn't touch my hosts and other important
> custom files, right?

Your diff would have caught it. I scrolled past it too fast and zeroed 
in on other things :).

>
> > I would also go to the trouble of tee'ing the makes and
> > create logs of
> > the makes and installs. Since you are going on, you want to know
> > what you have done. What I typically do is
> >
> > make     buildworld 2>&1 | tee /var/log/build/bworld-`date
> > "+%Y%m%d-%H%M"`.log
>
> Excellent idea, if I was tight on disk space, I could even gzip them.
>
> > required minutes in order to be unique. I also log all of my
> > cvsups.
>
> Another great idea!  Thanks!
>
> I've just been working on timing the different processes, here's a
> snippet...
>
> Printf "Time for buildworld:\t" >> /root/build.log
> \time -aho /root/build.log make buildworld
> Printf "\n" >> /root/build.log
>
> BTW, which field do I want from the output of time?  Real, user,
> whatever?
> I'll also need to do some cut'ing in order to just get the proper
> field from time...or I could just let it rip...?
> For instance...
>
> Time for buildworld:  2:34:15 real	2:35:12 user	etc...

When I started timing my buildworlds, I figured out that you need 
everything but the right hand stuff in the following

AMD Athlon 2000+ XP
877.636u 233.835s 23:02.33 80.4%        1350+1662k 46008+7469io 
2359pf+0w
bw w -j2
891.151u 303.327s 35:02.77 56.8%        1305+1600k 52256+138679io 
2749pf+0w
bw w -j3
894.429u 305.373s 31:29.87 63.4%        1305+1595k 50306+138970io 
1914pf+0w
bw w -j4
893.259u 310.180s 32:43.57 61.2%        1308+1597k 56102+138705io 
2145pf+0w
bw w -j5
893.563u 311.353s 30:31.44 65.7%        1302+1591k 53927+138930io 
2331pf+0w

The wall clock time is really how efficient your process is. The fact 
that your user time or sys time is faster doesn't mean much if the wall 
clock time is 1/3 longer. 

I haven't automated timing this part because I time the whole script and 
not just the build. I would cut off the right hand side but it is too 
trivial to use just "time mkworld" and cut and paste the time into a 
"cat >> times.log" of my builds. This also lets me add comments in 
addition to the build time information. For example, the AMD 2000+ was 
important because that was the first build after an upgrade from an 
Intel P III 866. The upgrade was needed because the old mobo went 
flaky.

I think that once you have it setup, I will probably make my KISS simple 
script more complicated :). I have one machine that the mouse doesn't 
work in single user mode and including the timing into my mkworld 
script would save time. I can always go back and edit my time log to 
include the comments. If I have a choice, I would add the date field to 
the right hand side so that I can connect the time to the build log.

Kent

-- 
Kent Stewart
Richland, WA

http://users.owt.com/kstewart/index.html