kern/153520: Boot from GPT ZFS root on HP BL460c G1 unstable

[Vladimir I. Pushkar]

>Number:         153520
>Category:       kern
>Synopsis:       Boot from GPT ZFS root on HP BL460c G1 unstable
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Dec 29 10:40:06 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator:     Vladimir I. Pushkar
>Release:        8.2-PRERELEASE
>Organization:
Digital Ventures LLC
>Environment:
FreeBSD temp.digital-ventures.net 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #0: Wed Dec 22 09:51:31 UTC 2010     root at temp.dvdev.org.ua:/usr/obj/usr/src/sys/GENERIC  amd64

>Description:
FreeBSD 8.2 installed using Fixit# environment on several HP ProLiant BL460c G1 servers with HP SmartArray mirror sometimes does not boot from ZFS pool. Installation done according to http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot with swap volume located on ZFS.
There are 2 disks configured on HP SmartArray as RAID1 array. Thus there is one logical device da0 and two GPT partitions:
# gpart show
=>       34  143305853  da0  GPT  (68G)
         34        128    1  freebsd-boot  (64K)
        162  143305725    2  freebsd-zfs  (68G)

# dmesg
ciss0: <HP Smart Array E200i> port 0x4000-0x40ff mem 0xfde80000-0xfdefffff,0xfde70000-0xfde77fff irq 18 at device 8.0 on pci11
ciss0: PERFORMANT Transport
ciss0: got 2 MSI messages]
ciss0: [ITHREAD]
da0 at ciss0 bus 0 scbus0 target 0 lun 0
da0: <COMPAQ RAID 1(1VOLUME OK> Fixed Direct Access SCSI-5 device 
da0: 135.168MB/s transfers
da0: Command Queueing enabled
da0: 69973MB (143305920 512 byte sectors: 255H 32S/T 17562C)
# diskinfo -v da0
da0
        512             # sectorsize
        73372631040     # mediasize in bytes (68G)
        143305920       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        17562           # Cylinders according to firmware.
        255             # Heads according to firmware.
        32              # Sectors according to firmware.
        P688MQ0598      # Disk ident.
# pciconf -lv
ciss0 at pci0:11:8:0:      class=0x010400 card=0x3211103c chip=0x3238103c rev=0x00 hdr=0x00
    vendor     = 'Hewlett-Packard Company'
    device     = 'Smart Array E200/E200i SAS/SATA Controller'
    class      = mass storage
    subclass   = RAID
# zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
zroot                      12,0G  55,0G   548M  legacy
zroot/home                   56K  55,0G    56K  /home
zroot/storage                21K  30,0G    21K  /storage
zroot/swap                   10G  65,0G    16K  -
zroot/usr                  1,31G  55,0G   760M  /usr
zroot/usr/obj                20K  55,0G    20K  /usr/obj
zroot/usr/ports             267M  55,0G   267M  /usr/ports
zroot/usr/ports/distfiles    20K  55,0G    20K  /usr/ports/distfiles
zroot/usr/ports/packages     21K  55,0G    21K  /usr/ports/packages
zroot/usr/src               311M  55,0G   311M  /usr/src
zroot/var                   125M  55,0G   247K  /var
zroot/var/db                125M  55,0G   121M  /var/db
zroot/var/db/pkg           3,94M  55,0G  3,94M  /var/db/pkg
zroot/var/log               215K  55,0G   215K  /var/log
zroot/var/mail               20K  55,0G    20K  /var/mail

# md5 /boot/gptzfsboot 
MD5 (/boot/gptzfsboot) = fd31f87e1c8e2bb4fa06b89658c63d06

# zdb -uuu zroot
Uberblock

        magic = 0000000000bab10c
        version = 15
        txg = 20726
        guid_sum = 3529677120116096247
        timestamp = 1293618442 UTC = Wed Dec 29 12:27:22 2010
        rootbp = [L0 DMU objset] 800L/200P DVA[0]=<0:1200ee400:200> DVA[1]=<0:3803ec200:200> DVA[2]=<0:5c0005c00:200> fletcher4 lzjb LE contiguous birth=20726 fill=245 cksum=c37f85b6c:4b1d45d93d8:eb0f06388e48:1f34f5d644dceb


Sometimes, after high disk load (e.g. buildworld) followed by reboot we got:
error 1 lba 32
error 1 lba 1
No ZFS pools located, can't boot

But after some numbers of reboots we can normally boot...

>How-To-Repeat:
1. Install 8.2-RC1 amd64 according to http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot
2. reboot
3. rebuild/install world && kernel
4. reboot

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: