IPMI serial console

[Jeremy Chadwick]

On Fri, Feb 22, 2013 at 10:51:17AM +1030, Daniel O'Connor wrote:
> 
> On 22/02/2013, at 9:59, Jeremy Chadwick <jdc at koitsu.org> wrote:
> > The reason I've advocated use of -Sxxx in /boot.config for years is
> > because it gets around whatever idiocy there is in the FreeBSD kernel
> > pertaining to serial port speed limitation.  Possibly those boot2
> > changes I mentioned above have since dealt with this, but the situation
> > used to be that without -Sxxx in /boot.config, you had to set
> > BOOT_COMCONSOLE_SPEED=115200 in /etc/make.conf (or during buildworld and
> > buildkernel) to achieve a serial port speed >9600bps -- otherwise
> > FreeBSD would limit/cap the serial speed to 9600bps and you wouldn't see
> > any output (even once getty(8) started -- yet switching the serial
> > connection to 9600bps instantly made things work).  Like I said,
> > possibly this has been dealt with.
> 
> Hmm I tried putting '-S 115200' in /boot.config and it broke - the boot process didn't run the loader (or kernel).

I'll talk a bit about this -- again, sorry for the verbosity.  I'll
explain what I've historically used/done, then speculate a bit about
your IPMI stuff:

For me, on systems without IPMI, all I had to do was this (and nothing
else):

* Put the following in /boot.config:

-S115200 -Dh

* Within the PC BIOS, enable VGA-to-serial redirection, setting the
serial port to 115200, 8-N-1, COM1 (if I remember right for our boards),
and hardware flow control enabled.  I also remember enabling an option
called something like "Retain POST Hand-off" (I can dig up the actual
BIOS option name if you want), which causes the INT 0x10 interrupt
vector re-route for BIOS output to be retained ***after*** POST.

All the VGA-to-serial redir allows is the ability to go into the BIOS
via the serial port (by pressing Ctrl-? (Backspace or Del depending on
your terminal setup)), see the BIOS boot-up via serial, option ROM boot
messages, PXE option ROM messages, and so on.

That BIOS option whose name I forget also allows you to see messages
past that point (as long as code being run doesn't stomp over the
interrupt vectors).  Thus with boot0 (rather than boot0sio -- I've never
used boot0sio) this allows you to see any boot0 related messages (if
any) due to VGA-serial redirection.  boot0 is the classic 512-byte
stuff, so there isn't much room to tinker there.

boot1/boot2 is what loads /boot.config and outputs messages like
"/boot.config: -S115200 -Dh" on (if my memory serves me right) both
serial and VGA console.  It's also responsible for the ">> FreeBSD/i386
BOOT" prompts, which you can induce if you hit a key (serial or
physical) very early on in the boot process.

boot2 is, if I remember right, what actually ends up doing a bunch of
other "stuff" that can often anger serial ports or "redirection" bits
that a BIOS may be doing behind the scenes.

Things are then handed off to loader (the big FreeBSD logo menu, etc.),
which also has its own set of variables which you can set for serial
console/etc. -- but are not necessary if you use /boot.config, since
boot1/boot2 pass on to loader the fact that -Dh is being used, and thus
loader should not mess with anything.

Then loader loads xxx_load modules and the kernel, and starts the
kernel.  The kernel, before any output is printed (hence my request for
a verbose boot (thinking it might print something beforehand)), does a
whole bunch of magic which (for lack of better term) "stomps over" all
of the previous bootloaders' work.  There is a hand-off between loader
and the kernel regarding serial console usage.  I never saw kernel
output on the VGA console at this point -- only serial; the FreeBSD
Handbook documents this fact.  You will, however, on VGA console see the
cursor change shape (changing the cursor mask) at some point.

Finally init(8) gets run, which in turn uses ttys(5), thus getty(8) gets
run to adjust the terminal parms on /dev/ttyuX, speed, etc. and bring up
a "login:" prompt for the serial console.  The End.

Now for you...

Your situation may be different because you have 3 serial ports (2
classic DB9 ports or headers, and one "fake" via IPMI), so you may need
to rely entirely on /boot/loader.conf to accomplish use of the IPMI one,
unless you wanted to set BOOT_COMCONSOLE_PORT.

I believe boot2 assumes 0x3f8 as the serial console I/O port (unless
BOOT_COMCONSOLE_PORT is redefined per make.conf etc.), so -S would be
adjusting the speed of that.  I'm still not sure how boot2/etc. figures
out what IRQ to use for an alternate port (betting there's some PC
architecture thing which I've forgotten that can return this), since in
your case the IPMI serial port is 0x3e8, IRQ 5.

Possibly the reason you see via the IPMI serial port at this stage is
because IPMI also does VGA-to-IPMI output, so what you're seeing on the
IPMI serial port is actually from the VGA console (speculation on my
part).

So there are many "phases" of the whole thing where something can go
wrong.  It sounds to me like the issue is either in boot2 or the loader
hand-off to the kernel.  Navdeep had some ideas, which he commented he
misunderstood, but I think he's going in the right direction.

Debugging all of this is as you know a PITA because of where during the
whole startup process it lies.  IPMI just makes this whole thing an even
bigger mess because it ties itself in to bits/pieces along the way,
which a kernel (or even a bootloader, depending on what it touches and
how), can mess up.  This is why I've always stuck with the classic DB9
serial ports on the backplane; I know how to get FreeBSD to behave right
with those, everything else is voodoo.  :-)

Part of me wonders if it's possible to disable, say, COM1 in the BIOS,
then in the IPMI firmware tell it to user a serial I/O port of 0x3f8,
IRQ 4 (i.e. COM1) and see if that works with the method I describe
above.  I don't have much familiarity with IPMI by choice, solely
because of situations exactly like what you're going through.  I have
the same opinion of those damn NIC ASF things (see bge(4)) -- which is
exactly why many motherboard vendors that do IPMI now offer a
*physically separate NIC/RJ45 port* for it, rather than "piggybacking":
the latter caused so much pain/anger that it wasn't worth it.

> > For Daniel: have you tried a verbose boot, to see if you get *anything*
> > prior to the initial "Copyright" line?  (Keep a raw I/O dump too, to see
> > if ANY characters are received, not just printable ones -- that can
> > sometimes help determine if some code is initialising something wrong)
> 
> I had a look by running ipmitool inside script but I could not see any
> characters after the BIOS prints a countdown to boot.

Good to know -- thank you.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |