Speed up booting on 10-stable?

[Gabor Radnai]

Hi,

I have a strange problem / behavior  please let me know if this normal or
something I can do about it.

I've the following specs hw running 10-stable (r268038): details at the
bottom.

Issue is that booting and getting prompt takes around 3,5 minutes from
power on. As I understand booting has several stages times are as follows (
start -> end ):
power on + BIOS initialization 0 -> 25 sec
BTX loader 25 -> 40
bootstrap loader 40 -> 57
Welcome to menu 57 -> 1 min 25s
loading kernel 1:25 -> 1:59
Booting 1:59 -> 2:24
Prompt 2:24 -> 3:30

I appreciate some phases are due to hw (e.g. BIOS initialization) or custom
config (e.g. loading all my stuff after Booting message) but all other
seems very slow to expected throughput of the mirrored SSD boot setup I
have.

Both SSD is the same capable of reading ~ 400MB/s, kernel + modules (aio
zfs opensolaris aesni crypto cryptodev) size is <25M why it takes almost 30
sec to read for example?

And the strangest thing is that if i use a Windows host to guest a virtual
FreeBSD everything is much faster. Getting BTX, boot strap, kernel loading
it is just couple of seconds, to  "Booting..." is less than 20 sec from
power on (versus on hw ~ 120 sec).

Again, I appreciate some part of the boot process takes what it takes but
loading BTX loader till getting 'Booting ...' message I feel being
abnormally very slow.

Any advise / comment is welcomed.

Thank you.

--------------------- Specs -----------------------

Supermicro X9SCM-F

CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3200.10-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206a7  Family = 0x6  Model = 0x2a
Stepping = 7

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics

real memory  = 8598323200 (8200 MB)

ada0: <SAMSUNG SSD 830 Series CXM03B1Q> ATA-9 SATA 3.x device
ada0: Serial Number S0Z2NSAC808513
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
ada1: <SAMSUNG SSD 830 Series CXM03B1Q> ATA-9 SATA 3.x device
ada1: Serial Number S0VXNYABC04893
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad6


Am using ZFS on root made as mirror of ada0 + ada1
=>       34  125045357  ada0  GPT  (60G)
         34          6        - free -  (3.0K)
         40       1024     1  boot0  (512K)
       1064        984        - free -  (492K)
       2048    4194304     2  swap0  (2.0G)
    4196352   41943040     3  sys0  (20G)
   46139392    4194304     4  log0  (2.0G)
   50333696   52428800     5  cache0  (25G)
  102762496   22282895        - free -  (11G)

=>       34  125045357  ada1  GPT  (60G)
         34          6        - free -  (3.0K)
         40       1024     1  boot1  (512K)
       1064        984        - free -  (492K)
       2048    4194304     2  swap1  (2.0G)
    4196352   41943040     3  sys1  (20G)
   46139392    4194304     4  log1  (2.0G)
   50333696   52428800     5  cache1  (25G)
  102762496   22282895        - free -  (11G)

  pool: sys
 state: ONLINE
  scan: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        sys           ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            gpt/sys0  ONLINE       0     0     0
            gpt/sys1  ONLINE       0     0     0

zpool is aligned (ashift=12) and filesystem has these non-default values:

NAME  PROPERTY     VALUE      SOURCE
sys   atime        off        local
sys   compression  lz4        local
sys   checksum     fletcher4  local