Puzzled: install system from hard disk to another does not work, but dvd install works

[Heikki Suonsivu]

I have traditionally done disk to disk installs, for example, when 
changing partition setup.  I think this worked with 7 and 8, but now 
with 9.* it seems to fail.  Now, this seems to be quite strange, and I 
cannot quite figure this out:

Setup is like this:

- host with two disks, shown up as ada0 and ada1

- I install FreeBSD 9.0R into ada0, with manual setup, which includes
     - gpt partitioning, with freebsd-boot partition at 64, size 192 
(aligned for 4kB blocks), freebsd-swap, and freebsd-ufs partitions
     - gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0
     - gmirror label root ada0p3 (other disk will be inserted later)
     - gjournal label /dev/mirror/root
     - newfs -J /dev/mirror/root.journal
     - And appropriate loader.conf etc setups

- This install works fine.

- Upgrade to latest using freebsd-update

- Then I notice that swap (or whatever) was too small, so lets redo the 
partitions on the second disk with changed partitions:

- Boot single user from the working system installed above

- Do the same partitioning steps as earlier from dvd

- tar/rsync whatever all the files from ada0 to ada1

- reboot from ada1.  Now, this hangs when it tries to mount root, the 
last message is

   Trying to mount root from ufs:/dev/mirror/root.journal 
[rw,async,noatime]...

   And after this nothing happens (at least for hours).

So, what might be wrong?  Further debugging:

- If I remove journal and mirror (not changing anything else), it still 
stops when trying to mount root, so it does not seem that mirror and 
journal have effect (though I have not tried if ordinary partition to 
partition copy works)

- noatime option on mount does not have effect on this

- Redoing bootcode commands does not cure it

- This is not specific to computer or OS version.  I have had same 
problem on modern amd64 host as well as old i386 host, and this has 
repeated on at least 4 systems, all different.  This happened with 
earlier 9.0R versions (binary updated) as well, though I cannot be 
certain if unmodified 9.0R has this problem.  Medias have been anything 
from various sized disks to CF cards.  I think doing disk-to-disk 
install worked with 8.x (though I was mostly using mbr partitions that 
time).

- Media size does not seem to have any effect.

- When I give the computer three-letter salute (ctrl-alt-del), it stops 
the journals and mirrors, and reboots normally.  So the kernel is not 
completely hung, and shuts down properly.

- The root disk does not seem to have gotten mounted at all, as dumpfs 
says "last mounted at" to be /mnt, not / (it was mounted to /mnt when I 
tried this)

- Contents of /boot is identical on both disks (as are all other files, 
copied with either tar or rsync).

- Contents of gpt freebsd-boot partitions are identical

- There was slight difference in mbr block, but copying ada0 mbr block 
to ada1 makes no difference.  The difference was that partition data on 
ada1 was different, with very small partition at the end.  Reinstalling 
pmbr with gpart bootcode does not have effect.

- fsck passes without errors.

- Dumpfs seems to have no strange differences (other than slightly 
different size filesystem

- Doing this from multi user or single user system does not make a 
difference

- Having disk to install from in single user mode as read-only does not 
make a difference (can't use vi to edit fstab but that is not strictly 
necessary...)

- Repeating all steps does not make a difference

- I have done this so many times, that I am pretty sure I am not typoing 
something every time.

- If I drop gjournal and mirror and create filesystem with softupdates 
journaling, the resulting system boots

- If I have mirror with no gjournal, and softupdates journaling (tested 
by labeling already existing partition in previous test, so it might not 
be completely similar situation).

Apart from changed partition sizes, there is no visible difference in 
boot blocks, partition tables, filesystems, and files on the 
partitions.   The commands to create the partitions from dvd seem 
completely identical, there is no visible difference.  Kernel just waits 
for something when trying to mount root filesystem on ada1, while boot 
from ada0 works fine.

Any idea what is going on?

Has anyone else tried this?

In what situations root mount could get stuck waiting for something?

Heikki Suonsivu
bbnetworks.net