Diskless boot fails when network card is reset before NFS root mount

[Daniel Feenberg]

Our dozen diskless FreeBSD 7.0 machines are all able to diskless boot just 
fine. However, when we tried to set up a FreeBSD 8.0 root for them to boot 
from, the boot process would load up all the devices, and then fail right 
after the line "NFS ROOT: ..."

We boot using pxeboot. pxeboot then mounts our NFS root and runs the 
loader from /boot under there. After the beastie screen, loader runs the 
kernel. All this works fine under both 7 and 8.

On both 7 and 8, we see messages of the form "em0: link state changed to 
down","em0: link state changed to up". They happen right before or after 
NFS ROOT. Then, on version 8, we see error messages about /devfs not being 
found, and eventually /sbin/init not being found.

We surmise that what happens is that the kernel resets the ethernet 
interface, right before re-mounting the NFS root (note that the NFS root 
was already mounted back before the beastie screen). On 8, somehow the 
interface reset interferes with the nfs mount resulting in no root FS.

This problem seems to be referred to here:

   http://lists.freebsd.org/pipermail/freebsd-net/2009-January/020666.html

Have others seen this issue? Is it a known bug? Is there a workaround or a 
fix?

It seems to us, not being kernel hackers, a particularly difficult problem 
to get a handle on because execution is being controlled by the kernel at 
that point, there is no loader script or rc script that one could insert 
debugging print statements into.

A complete description of our diskless boot procedure is given at:

   http://www.nber.org/sys-admin/FreeBSD-diskless.html

which has worked well on several prior versions of FreeBSD. The network 
card is an Intel Pro/1000 card - well supported by FreeBSD.

   - Alex Aminoff
     BaseSpace.net
     National Bureau of Economic Research (nber.org)

   - Daniel Feenberg
     feenberg at nber.org
     National Bureau of Economic Research (nber.org)