upgrade from 6.3 to 7.0 problems

[Cam]

Hello,

I have a botched upgrade from 6.3-release to 7.0-release on my hands. 
The 'make buildworld', 'make buildkernel' and 'make installkernel' went 
fine.  'Make installworld' did not - I think I forgot to drop into 
single-user mode but did not keep good notes.

uname -a:
FreeBSD server.shiner.ca 7.0-RELEASE-p5 FreeBSD 7.0-RELEASE-p5 #0: Tue 
Oct 14 10:16:32 CDT 2008 
root at server.shiner.ca:/usr/obj/usr/src/sys/GENERIC  i386

Here is a hint of the problem (there is a bad duplex to clean up , which 
is my next task):
server# gmirror remove gm0 ad3
Userland and kernel parts are out of sync.


I cannot install the world in single-user mode:

# cd /usr/src
# env -i make installworld
mkdir -p /tmp/install.9f0tXCFc
for prog in [ awk cap_mkdb cat chflags chmod chown  date echo egrep find 
grep install-info  ln lockf make mkdir mtree mv pwd_mkdb rm sed sh 
sysctl  test true uname wc zic; do  cp `which $prog` 
/tmp/install.9f0tXCFc;  done
cd /usr/src; MAKEOBJDIRPREFIX=/usr/obj  MACHINE_ARCH=i386  MACHINE=i386 
  CPUTYPE=  GROFF_BIN_PATH=/usr/obj/usr/src/tmp/legacy/usr/bin 
GROFF_FONT_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/groff_font 
GROFF_TMAC_PATH=/usr/obj/usr/src/tmp/legacy/usr/share/tmac 
PATH=/usr/obj/usr/src/tmp/legacy/usr/sbin:/usr/obj/usr/src/tmp/legacy/usr/bin:/usr/obj/usr/src/tmp/legacy/usr/games:/usr/obj/usr/src/tmp/usr/sbin:/usr/obj/usr/src/tmp/usr/bin:/usr/obj/usr/src/tmp/usr/games:/tmp/install.9f0tXCFc 
make -f Makefile.inc1 reinstall
--------------------------------------------------------------
 >>> Making hierarchy
--------------------------------------------------------------
[snipped, no errors]
--------------------------------------------------------------
 >>> Installing everything
--------------------------------------------------------------
cd /usr/src; make -f Makefile.inc1 install
===> share/info (install)
===> lib (install)
===> lib/csu/i386-elf (install)
/usr/local/libexec/ccache/world-cc -O -pipe 
-I/usr/src/lib/csu/i386-elf/../common 
-I/usr/src/lib/csu/i386-elf/../../libc/include -Wsystem-headers -Wall 
-Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes 
-Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual 
-Wwrite-strings -Wswitch -Wshadow -Wcast-align -Wunused-parameter 
-Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls 
-Wno-pointer-sign -c crt1.c
*** Error code 1
Stop in /usr/src/lib/csu/i386-elf.
*** Error code 1
Stop in /usr/src/lib.
*** Error code 1
Stop in /usr/src.
*** Error code 1
Stop in /usr/src.
*** Error code 1
Stop in /usr/src.
*** Error code 1


I tried to rebuild the world from scratch by following these 
instructions from handbook section 24.4:

chflags –R noschg /usr/obj/usr
rm –rf /usr/obj/usr
cd /usr/src
make cleandir
make cleandir
cd /usr/src/usr.sbin/mergemaster
./mergemaster.sh –p
cd /usr/obj
chflags –R noschg *
rm –rf *
cd /usr/src
env –i make buildworld
env –i make -DALWAYS_CHECK_MAKE buildkernel
env –i make -DALWAYS_CHECK_MAKE installkernel
<reboot in single-user>
fsck –p
mount –u /
mount –a –t ufs
swapon –a
adjkerntz –i
cd /usr/src
env –i make installworld
mergemaster
reboot


This is the result:

< everything ok up to this step>
server#    env -i make buildworld
--------------------------------------------------------------
 >>> World build started on Tue Oct 14 11:42:00 CDT 2008
--------------------------------------------------------------
--------------------------------------------------------------
 >>> Rebuilding the temporary build tree
--------------------------------------------------------------
[snipped, no errors]
--------------------------------------------------------------
 >>> stage 1.1: legacy release compatibility shims
--------------------------------------------------------------
cd /usr/src; MAKEOBJDIRPREFIX=/usr/obj/usr/src/tmp  INSTALL="sh 
/usr/src/tools/install.sh" 
PATH=/usr/obj/usr/src/tmp/legacy/usr/sbin:/usr/obj/usr/src/tmp/legacy/usr/bin:/usr/obj/usr/src/tmp/legacy/usr/games:/sbin:/bin:/usr/sbin:/usr/bin 
  WORLDTMP=/usr/obj/usr/src/tmp  MAKEFLAGS="-m /usr/src/tools/build/mk 
-m /usr/src/share/mk" make -f Makefile.inc1  DESTDIR= 
BOOTSTRAPPING=602000  -DWITHOUT_HTML -DWITHOUT_INFO -DNO_LINT 
-DWITHOUT_MAN  -DWITHOUT_NLS -DNO_PIC -DWITHOUT_PROFILE -DNO_SHARED 
-DNO_CPU_CFLAGS -DNO_WARNS legacy
===> tools/build (obj,includes,depend,all,install)
/usr/obj/usr/src/tmp/usr/src/tools/build created for /usr/src/tools/build
cd /usr/src/tools/build; make buildincludes; make installincludes
rm -f .depend
CC='/usr/local/libexec/ccache/world-cc' mkdep -f .depend -a 
-I/usr/obj/usr/src/tmp/legacy/usr/include /usr/src/tools/build/dummy.c
mkdep: compile failed
*** Error code 1
Stop in /usr/src/tools/build.
*** Error code 1
Stop in /usr/src.
*** Error code 1
Stop in /usr/src.
*** Error code 1
Stop in /usr/src.


Blech.  I've floundered about without success.  How can I clean up this
mess?  All comments and thoughts welcome.  I have not tried a binary
upgrade.  Will re-install but as a last option.

Many thanks,
Cam