IPMI serial console

[Jeremy Chadwick]

On Fri, Feb 22, 2013 at 01:00:01AM +0200, Konstantin Belousov wrote:
> On Fri, Feb 22, 2013 at 09:18:51AM +1030, Daniel O'Connor wrote:
> > 
> > On 22/02/2013, at 9:15, Navdeep Parhar <nparhar at gmail.com> wrote:
> > >> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 on acpi0
> > >> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
> > >> uart2: <16550 or compatible> port 0x3e8-0x3ef irq 5 flags 0x30 on acpi0
> > >> 
> > >> The loader talks on the serial console fine, it's the kernel that doesn't use it which is the problem.
> 
> It might be not the serial port, to which the loader talks. The supermicro
> boards I dealt with, have a feature of VGA text mode redirection to the
> serial port. This is how bios redirection usually works.

As I understand it, the way this works is that the BIOS itself has code
that sets the INT 0x10 (BIOS) vector to point to some of its own code,
which then draws to the classic VGA framebuffer (segment 0xA000 if I
remember right) as well as output characters to the serial port whenever
something uses INT 0x10 ah=0x13 or ah=0x0e (or something along these
lines -- pardon me if I have the function numbers wrong, it's been quite
some time).

Messing with those vectors, and/or touching some other things (such as
initialising serial port bits in a certain way) can cause this
functionality to stop working right then and there -- the assumption is
that whatever just touched it did so Correctly(tm) and has set up its
own code/vectors to handle the serial + VGA output.

The FreeBSD bootstraps have had some changes made to them in the past
5-6 years (I can try to dig up the commits if you want, there were many)
that "tweak" some of this initialisation, in attempt to minimise the
amount of adjustments being made to certain bits/things (specifically to
keep BIOS-level VGA-to-serial redirection from getting stomped on).  I
believe both boot0 and boot2/loader have been tweaked in this regard.

The kernel, on the other hand, may not be doing the Right Thing(tm) once
control is handed off to it.  The kernel does have some knowledge of the
hand-offs between boot2/loader and itself, as you know, but it sounds to
me like the kernel may be tweaking something that upsets a working IPMI
serial redirection.

IPMI makes this situation even weirder, since it introduces its own
separate serial port and other nonsense.  This is one of many reasons I
always avoided IPMI -- it's one of those "sounds great" inventions, but
in practise is a fucking nightmare.  When it works (reliably) it's
great, otherwise prepare to pull your hair out.  For example, the folks
I knew at Yahoo in Sunnyvale who had to deal with IPMI serial console
across hundreds of systems had to actually bring Supermicro engineers
*to their campus* to show them the issues + get fixes (in the IPMI
firmware itself).  I'm not picking on Supermicro BTW, just to be clear.

> You could look at some bios knob which controls the point where the
> said redirection is stopped. It should be like 'after the OS takes
> the control', and not 'forever'. For BIOS, the loader is OS.

That would be BIOS-level stuff, not IPMI-level stuff.  There's definitely
a difference.

What Daniel is effectively wanting is for the FreeBSD bootstraps to use
serial port I/O address 0x3e8, IRQ 5, and initialise a serial port speed
of 115200 -- and the kernel to use that too.

He says that the bootstraps show text output from boot0 (assuming he's
using -Dh -S115200 in /boot.config), and from boot2/loader too.  It's
once the kernel starts that he sees no output.  Not until getty(8) is
loaded via ttys(5) does he get output (he gets a login: prompt).  It's
as if the kernel is pretending there's no serial console.

The reason I've advocated use of -Sxxx in /boot.config for years is
because it gets around whatever idiocy there is in the FreeBSD kernel
pertaining to serial port speed limitation.  Possibly those boot2
changes I mentioned above have since dealt with this, but the situation
used to be that without -Sxxx in /boot.config, you had to set
BOOT_COMCONSOLE_SPEED=115200 in /etc/make.conf (or during buildworld and
buildkernel) to achieve a serial port speed >9600bps -- otherwise
FreeBSD would limit/cap the serial speed to 9600bps and you wouldn't see
any output (even once getty(8) started -- yet switching the serial
connection to 9600bps instantly made things work).  Like I said,
possibly this has been dealt with.

Sorry for my rambling Email, but there's a lot of history and crap
contained here (almost 10 years worth) that's hard to cover tersely.

For Daniel: have you tried a verbose boot, to see if you get *anything*
prior to the initial "Copyright" line?  (Keep a raw I/O dump too, to see
if ANY characters are received, not just printable ones -- that can
sometimes help determine if some code is initialising something wrong)

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |