[REGRESSION] Root zpool mounting broken between 06/30/2013 and 07/21/2013 when PS/2 support compiled into the kernel

[Garrett Cooper]

On Mon, Jul 28, 2014 at 9:37 AM, John Baldwin <jhb at freebsd.org> wrote:
>
> On Jul 28, 2014, at 9:07 AM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
>
>> On Jul 22, 2013, at 10:58 AM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
>>
>>> On Jul 22, 2013, at 9:08 AM, John Baldwin <jhb at freebsd.org> wrote:
>>>
>>>> On Monday, July 22, 2013 10:30:32 am Garrett Cooper wrote:
>>>>> I have a KERNCONF that previously had PS/2 support compiled into the kernel. If I comment out the following lines like so:
>>>>>
>>>>> # atkbdc0 controls both the keyboard and the PS/2 mouse
>>>>> #device         atkbdc          # AT keyboard controller
>>>>> #device         atkbd           # AT keyboard
>>>>>
>>>>> then I'm able to mount root again (it was failing with ENOXDEV).
>>>>>
>>>>> The working kernel was as follows:
>>>>>
>>>>> $ strings /boot/kernel.WORKING/kernel | grep -B 2 -A 2 BAYONETTA
>>>>> @(#)FreeBSD 9.1-STABLE #7 r+0304216: Sun Jun 30 15:22:55 PDT 2013
>>>>> FreeBSD 9.1-STABLE #7 r+0304216: Sun Jun 30 15:22:55 PDT 2013
>>>>>  gcooper at bayonetta.local:/usr/obj/scratch/git/github/yaneurabeya-freebsd-stable-9/sys/BAYONETTA
>>>>> gcc version 4.2.1 20070831 patched [FreeBSD]
>>>>> FreeBSD
>>>>> 9.1-STABLE
>>>>> BAYONETTA
>>>>> $ cd /usr/src; git log 0304216
>>>>> commit 03042167f73c213732b44218a24d8e1bbea00f8c
>>>>> Merge: 2edcad2 974abfb
>>>>> Author: Garrett Cooper <yanegomi at gmail.com>
>>>>> Date:   Mon Jun 24 19:00:45 2013 -0700
>>>>>
>>>>>  Merge remote-tracking branch 'upstream/stable/9' into stable/9
>>>>>
>>>>> The working kernel [with atkbdc] was as follows:
>>>>>
>>>>> FreeBSD bayonetta.local 9.2-BETA1 FreeBSD 9.2-BETA1 #12 r+c178034: Sun Jul 21 20:19:38 PDT 2013
>>>> root at bayonetta.local:/usr/obj/scratch/git/github/yaneurabeya-freebsd-stable-9/sys/BAYONETTA  amd64
>>>>> $ git log c178034
>>>>> commit c17803445f4ffb97e1a46a1be5f7ea04692793f0
>>>>> Author: avg <avg at FreeBSD.org>
>>>>> Date:   Tue Jul 9 08:30:31 2013 +0000
>>>>>
>>>>>  zfsboottest.sh: remove checks for things that are not strictly required
>>>>>
>>>>>  MFC after:  10 days
>>>>>
>>>>> (Yes, I had to backport some things because they are busted on stable/9 due to other incomplete/missing MFCs).
>>>>>
>>>>> I can test out patches, but I don't have time to bisect the actual commit that caused the failure. That being said my intuition says it's this
>>>> commit should be looked at first:
>>>>>
>>>>> commit 28f961058b0667841d7e9d8639bfd02ed8689faa
>>>>> Author: jhb <jhb at FreeBSD.org>
>>>>> Date:   Wed Jul 17 14:04:18 2013 +0000
>>>>>
>>>>>  MFC 252576:
>>>>>  Don't perform the acpi_DeviceIsPresent() check for PCI-PCI bridges.  If
>>>>>  we are probing a PCI-PCI bridge it is because we found one by enumerating
>>>>>  the devices on a PCI bus, so the bridge is definitely present.  A few
>>>>>  BIOSes report incorrect status (_STA) for some bridges that claimed they
>>>>>  were not present when in fact they were.
>>>>>
>>>>>  While here, move this check earlier for Host-PCI bridges so attach fails
>>>>>  before doing any work that needs to be torn down.
>>>>>
>>>>>  PR:         kern/91594
>>>>>  Approved by:        re (marius)
>>>>
>>>> I strongly doubt that this is related.  It would be most helpful if you could
>>>> obtain a dmesg from the new kernel however (perhaps via a serial console) to
>>>> rule it out.  All you would need to see is if the new kernel sees more "pcib"
>>>> devices than the old one to see if this change even has an effect on your
>>>> system.
>>>
>>> Unfortunately the USB keyboard is broken as well at the mount root prompt and the workstation doesn't have a uart on it that I can play with (it's my home box), so I'm dead in the water when it panics at the mount root prompt right now.
>>>
>>> I guess I can revert this and a handful of other amd64/ata_cam/zfs commits to see if this goes away, but I won't be getting to that before next Sunday probably as this is my file server and DNS server now.
>>
>>       I ran into the issue going from vanilla 9.2-RELEASE-p10 to 9.3-RELEASE as well :(. I’ve filed this bug to track the issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192183 . I’ll see if GENERIC can boot my system sometime this week (the KERNCONF has been working for several releases, but it could be an issue with that that’s being overlooked by accident).
>> Thanks!

...

> Also, you would need to get verbose dmesg's of old and new kernels as a first step in narrowing it down.

I can't do trivial debugging because my USB keyboard doesn't work at
the mountroot prompt (
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=133989 ), I don't
have a PS/2 keyboard, and the system only has VGA access :(. I'll try
working on disabling the PS/2 controller in the BIOS and a few other
things to force the system to stop ignoring the USB keyboard to get
scrollback, because that appeared to work for some folks with this
issue according to the ukbd bug I referenced.

-Garrett