[REGRESSION] Root zpool mounting broken between 06/30/2013 and 07/21/2013 when PS/2 support compiled into the kernel

[John Baldwin]

On Monday 28 July 2014 09:57:08 Garrett Cooper wrote:
> On Mon, Jul 28, 2014 at 9:37 AM, John Baldwin <jhb at freebsd.org> wrote:
> > On Jul 28, 2014, at 9:07 AM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
> >> On Jul 22, 2013, at 10:58 AM, Garrett Cooper <yaneurabeya at gmail.com> 
wrote:
> >>> On Jul 22, 2013, at 9:08 AM, John Baldwin <jhb at freebsd.org> wrote:
> >>>> On Monday, July 22, 2013 10:30:32 am Garrett Cooper wrote:
> >>>>> I have a KERNCONF that previously had PS/2 support compiled into the
> >>>>> kernel. If I comment out the following lines like so:
> >>>>> 
> >>>>> # atkbdc0 controls both the keyboard and the PS/2 mouse
> >>>>> #device         atkbdc          # AT keyboard controller
> >>>>> #device         atkbd           # AT keyboard
> >>>>> 
> >>>>> then I'm able to mount root again (it was failing with ENOXDEV).
> >>>>> 
> >>>>> The working kernel was as follows:
> >>>>> 
> >>>>> $ strings /boot/kernel.WORKING/kernel | grep -B 2 -A 2 BAYONETTA
> >>>>> @(#)FreeBSD 9.1-STABLE #7 r+0304216: Sun Jun 30 15:22:55 PDT 2013
> >>>>> FreeBSD 9.1-STABLE #7 r+0304216: Sun Jun 30 15:22:55 PDT 2013
> >>>>> 
> >>>>>  gcooper at bayonetta.local:/usr/obj/scratch/git/github/yaneurabeya-freeb
> >>>>>  sd-stable-9/sys/BAYONETTA>>>>> 
> >>>>> gcc version 4.2.1 20070831 patched [FreeBSD]
> >>>>> FreeBSD
> >>>>> 9.1-STABLE
> >>>>> BAYONETTA
> >>>>> $ cd /usr/src; git log 0304216
> >>>>> commit 03042167f73c213732b44218a24d8e1bbea00f8c
> >>>>> Merge: 2edcad2 974abfb
> >>>>> Author: Garrett Cooper <yanegomi at gmail.com>
> >>>>> Date:   Mon Jun 24 19:00:45 2013 -0700
> >>>>> 
> >>>>>  Merge remote-tracking branch 'upstream/stable/9' into stable/9
> >>>>> 
> >>>>> The working kernel [with atkbdc] was as follows:
> >>>>> 
> >>>>> FreeBSD bayonetta.local 9.2-BETA1 FreeBSD 9.2-BETA1 #12 r+c178034: Sun
> >>>>> Jul 21 20:19:38 PDT 2013>>>> 
> >>>> root at bayonetta.local:/usr/obj/scratch/git/github/yaneurabeya-freebsd-st
> >>>> able-9/sys/BAYONETTA  amd64>>>> 
> >>>>> $ git log c178034
> >>>>> commit c17803445f4ffb97e1a46a1be5f7ea04692793f0
> >>>>> Author: avg <avg at FreeBSD.org>
> >>>>> Date:   Tue Jul 9 08:30:31 2013 +0000
> >>>>> 
> >>>>>  zfsboottest.sh: remove checks for things that are not strictly
> >>>>>  required
> >>>>>  
> >>>>>  MFC after:  10 days
> >>>>> 
> >>>>> (Yes, I had to backport some things because they are busted on
> >>>>> stable/9 due to other incomplete/missing MFCs).
> >>>>> 
> >>>>> I can test out patches, but I don't have time to bisect the actual
> >>>>> commit that caused the failure. That being said my intuition says
> >>>>> it's this>>>> 
> >>>> commit should be looked at first:
> >>>>> commit 28f961058b0667841d7e9d8639bfd02ed8689faa
> >>>>> Author: jhb <jhb at FreeBSD.org>
> >>>>> Date:   Wed Jul 17 14:04:18 2013 +0000
> >>>>> 
> >>>>>  MFC 252576:
> >>>>>  Don't perform the acpi_DeviceIsPresent() check for PCI-PCI bridges. 
> >>>>>  If
> >>>>>  we are probing a PCI-PCI bridge it is because we found one by
> >>>>>  enumerating
> >>>>>  the devices on a PCI bus, so the bridge is definitely present.  A few
> >>>>>  BIOSes report incorrect status (_STA) for some bridges that claimed
> >>>>>  they
> >>>>>  were not present when in fact they were.
> >>>>>  
> >>>>>  While here, move this check earlier for Host-PCI bridges so attach
> >>>>>  fails
> >>>>>  before doing any work that needs to be torn down.
> >>>>>  
> >>>>>  PR:         kern/91594
> >>>>>  Approved by:        re (marius)
> >>>> 
> >>>> I strongly doubt that this is related.  It would be most helpful if you
> >>>> could obtain a dmesg from the new kernel however (perhaps via a serial
> >>>> console) to rule it out.  All you would need to see is if the new
> >>>> kernel sees more "pcib" devices than the old one to see if this change
> >>>> even has an effect on your system.
> >>> 
> >>> Unfortunately the USB keyboard is broken as well at the mount root
> >>> prompt and the workstation doesn't have a uart on it that I can play
> >>> with (it's my home box), so I'm dead in the water when it panics at the
> >>> mount root prompt right now.
> >>> 
> >>> I guess I can revert this and a handful of other amd64/ata_cam/zfs
> >>> commits to see if this goes away, but I won't be getting to that before
> >>> next Sunday probably as this is my file server and DNS server now.>>> 
> >>       I ran into the issue going from vanilla 9.2-RELEASE-p10 to
> >>       9.3-RELEASE as well :(. I’ve filed this bug to track the issue:
> >>       https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192183 . I’ll
> >>       see if GENERIC can boot my system sometime this week (the KERNCONF
> >>       has been working for several releases, but it could be an issue
> >>       with that that’s being overlooked by accident).>> 
> >> Thanks!
> 
> ...
> 
> > Also, you would need to get verbose dmesg's of old and new kernels as a
> > first step in narrowing it down.
> I can't do trivial debugging because my USB keyboard doesn't work at
> the mountroot prompt (
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=133989 ), I don't
> have a PS/2 keyboard, and the system only has VGA access :(. I'll try
> working on disabling the PS/2 controller in the BIOS and a few other
> things to force the system to stop ignoring the USB keyboard to get
> scrollback, because that appeared to work for some folks with this
> issue according to the ukbd bug I referenced.

Do you have a serial port so you could use a serial console (or is this a 
laptop)?

-- 
John Baldwin