vbox + gpxe + pxeboot = fail [SOLVED]

[Lawrence Stewart]

On 07/10/10 17:04, Lawrence Stewart wrote:
> On 07/10/10 15:26, Lawrence Stewart wrote:
>> Hi All,
>>
>> I had some frustration trying to get FreeBSD to pxeboot inside a vbox VM
>> a while back. The thread is available here:
>>
>> http://lists.freebsd.org/pipermail/freebsd-emulation/2010-April/007681.html
>>
>> I left things for a while and came back to them yesterday with some
>> fresh resolve to nut the problem out. I have some new insights I wanted
>> to share.
>>
>> I'm using gpxe 1.0.1 from
>> http://kernel.org/pub/software/utils/boot/gpxe/ and doing the builds on
>> a Debian VM. To create a rom for the vbox AMD adapter types, I'm
>> following the details at:
>>
>> http://www.etherboot.org/wiki/romburning/vbox
>>
>> I turned the instructions to pad the rom into a python script you can
>> grab from here:
>>
>> http://people.freebsd.org/~lstewart/misc/vbox/rompad.py
>>
>>
>>
>> Here's what I've figured out so far:
>>
>> - The problem stems from the pxe boot rom environment provided by gpxe.
>> It sends and receives packets correctly, but somehow the IP addresses
>> get mangled (I think this happens inside gpxe) so it thinks the replies
>> it is waiting for should be coming in on one IP address when they
>> actually arrive on the real valid IP address.
>>
>> - Using the binary only vbox on Win XP which uses the Intel pxe boot rom
>> has no problems and works perfectly i.e. further evidence this is
>> isolated to gpxe
>>
>> - By changing the line
>> "if (udpread_p->status > 0) {"
>> to
>> "if (udpread_p->status > 1) {"
>> in sys/boot/i386/libi386/pxe.c, our pxeboot is able to work around the
>> problem and I can pxeboot FreeBSD just fine. gpxe therefore is correctly
>> reading the packets off the wire and passing them to our pxeboot code.
>> gpxe just sets the failure status code because it thinks the packet is
>> not the one we were waiting for because of the IP address being mismatched.
>>
>>
>>
>> The file in the gpxe distribution that I've been adding debug printf's
>> to is: src/arch/i386/interface/pxe/pxe_udp.c
>> By doing a "%s/DBG/dbg_printf/g" in that file, you get debugging output
>> that shows you the failures and the IP address it thinks the pkt should
>> be coming in on. In my case, it correctly sends UDP packets to
>> 172.16.7.21, and then waits for the reply on 172.16.7.50 (but sees that
>> the reply actually comes in on 172.16.7.43 which is the IP of the VM).
>> Because .43 != .50, gpxe returns status failure (i.e. 1) but does still
>> correctly read the pkt and pass it to our pxeboot hence why my hack of
>> ignoring the status actually allows things to work.
> 
> Some more useful info...
> 
> gpxe has the correct IP address at the beginning when printed from
> "pxenv_udp_open()". Then in FreeBSD's pxe_open() in
> sys/boot/i386/libi386/pxe.c, the "if (rootip.s_addr == 0) {" statement
> evaluates true so we do another DHCP exchange by doing "bootp(pxe_sock,
> BOOTP_PXE);" in order to try and get our own copy of the various DHCP
> variables required to NFS boot.
> 
> I think it's this second DHCP exchange that is somehow wiping gpxe's
> concept of the local IP as gpxe printfs done after the FreeBSD
> pxe_open() call return the bogus local IP. Sure enough, if I comment out
> the bootp() call and hardwire the rootip and rootpath, everything works
> as expected.
> 
> The reason "rootip.s_addr == 0" appears to be because the
> "pxe_call(PXENV_GET_CACHED_INFO);" in our pxeboot's pxe_init() fails to
> pull cached values from gpxe. I tried upping PXE_BUFFER_SIZE in case we
> weren't supplying a large enough buffer to receive all the cached data
> but that didn't make any difference.
> 
> So it appears there are two separate issues in our pxeboot's interaction
> with gpxe... first is we don't successfully manage to extract cached
> DHCP data from gpxe which causes us to do our own DHCP exchange in
> pxe_open() to get the info. Secondly, by doing the extra DHCP exchange,
> we're somehow trampling gpxe's concept of what the local IP is.

Please forgive the barrage of emails, but I've been in way over my head
and hoping for some enlightenment. After persisting a bit longer and
chatting with the helpful Doug White on IRC, I think I've got it all
nutted out. Here goes...

I did some more debugging and the "pxe_call(PXENV_GET_CACHED_INFO);" in
our pxeboot's pxe_init() doesn't fail like I previously speculated, and
in fact seems to work just fine and we end up with a copy of gpxe's
cached DHCP data in the "bootplayer" variable, which is of type
BOOTPLAYER defined as a struct in <sys/boot/i386/libi386/pxe.h>.

We then fall through into our pxe_open() and check "if (rootip.s_addr ==
0) {", and because rootip is always == 0, we fall into the "if". Why is
"rootip" always == 0 I hear you ask? Excellent question! Because we
never actually populate it (or any other necessary variables) with the
cached data we stashed in "bootplayer" back in pxe_init(). This is true
even in the case of the working Intel pxe boot rom.

So here's the tricky bit. In the gpxe case, the rom's DHCP request
includes a uid string of some bytes and my ISC DHCP server returns the
172.16.7.43 IP address to it. Then in the FreeBSD pxeboot binary we do
the unneeded bootp() call in pxe_open(), but we don't include the uid
value used by gpxe, so the DHCP server thinks this is a different client
and hands back a different IP address (you guessed it, 172.16.7.50). So
gpxe had IP .43, FreeBSD's boot loader updates it to .50 and everything
falls in a heap because gpxe thinks the packets inbound to .43 are bogus.

In the Intel pxe rom case, the rom doesn't include a uid in it's DHCP
request, so when FreeBSD's pxeboot code does the unneeded DHCP request,
it gets the same IP back and everything keeps working.

*phew*

Ok so what needs fixing...

1. FreeBSD's pxeboot code should try and use the cached data it gets
from the pxe boot rom up front. I think this means patching our
pxe_open() to fill in values from the "bootplayer" cache variable and
then only doing the bootp() call if the cached values are insufficient.

2. If in pxeboot we realise the cached data doesn't have all the options
we need to continue and we do have to do our own bootp() call to force a
new DHCP exchange, we should use the same uid (which I'm hoping is
stashed in the bootplayer variable - haven't confirmed yet) so that we
get the same IP back from the DHCP server that was assigned to the pxe
boot rom.

3. There is still the question of why packets are inbound to .43 at all
after our bootp() call gets the new IP address of .50. I'm unsure if
this is gpxe still sending packets using .43 but listening for packets
on .50 or if the FreeBSD pxeboot binary has cached the old IP and
continues to tell gpxe to use it. This needs to be tracked down. Step 2
may be unnecessary if we resolve this problem as things should be able
to happily switch over to the new IP address and keep working.

Cheers,
Lawrence