LSI 9240-4i 4K alignment

[George Kontostanos]

Hi all,

We have a server with a LSI 9240-4i controller configured in JBOD with
4 SATA disks. Running FreeBSD 9.1-Beta1:

Relevant dmesg:

FreeBSD 9.1-BETA1 #0: Thu Jul 12 09:38:51 UTC 2012
    root at farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3200.09-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206a7  Family = 6  Model = 2a  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16471670784 (15708 MB)
...
mfi0: <Drake Skinny> port 0xe000-0xe0ff mem
0xf7a60000-0xf7a63fff,0xf7a00000-0xf7a3ffff irq 16 at device 0.0 on
pci1
mfi0: Using MSI
mfi0: Megaraid SAS driver Ver 4.23
...
mfi0: 321 (397672301s/0x0020/info) - Shutdown command received from host
mfi0: 322 (boot + 3s/0x0020/info) - Firmware initialization started
(PCI ID 0073/1000/9241/1000)
mfi0: 323 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664
mfi0: 324 (boot + 3s/0x0020/info) - Firmware initialization started
(PCI ID 0073/1000/9241/1000)
mfi0: 325 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664
mfi0: 326 (boot + 5s/0x0020/info) - Package version 20.10.1-0107
mfi0: 327 (boot + 5s/0x0020/info) - Board Revision 03A
mfi0: 328 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3)
...
mfisyspd0 on mfi0
mfisyspd0: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd0:  SYSPD volume attached
mfisyspd1 on mfi0
mfisyspd1: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd1:  SYSPD volume attached
mfisyspd2 on mfi0
mfisyspd2: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd2:  SYSPD volume attached
mfisyspd3 on mfi0
mfisyspd3: 1907729MB (3907029168 sectors) SYSPD volume
mfisyspd3:  SYSPD volume attached
...
mfi0: 329 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) Info:
enclPd=ffff, scsiType=0, portMap=00,
sasAddr=4433221100000000,0000000000000000
mfi0: 330 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1)
mfi0: 331 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) Info:
enclPd=ffff, scsiType=0, portMap=02,
sasAddr=4433221102000000,0000000000000000
mfi0: 332 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2)
mfi0: 333 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) Info:
enclPd=ffff, scsiType=0, portMap=03,
sasAddr=4433221101000000,0000000000000000
mfi0: 334 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0)
mfi0: 335 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) Info:
enclPd=ffff, scsiType=0, portMap=01,
sasAddr=4433221103000000,0000000000000000
mfi0: 336 (397672376s/0x0020/info) - Time established as 08/07/12
16:32:56; (28 seconds since power on)

The problem:

When trying to create a RaidZ pool using gpart and perform a 4K
alignment using  gnop, we get the follwoing error immediately after
exporting the pool and destroying the .nop devices:

id: 8043746387654554958
  state: FAULTED
 status: One or more devices contains corrupted data.
 action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-5E
 config:

        Pool                      FAULTED  corrupted data
          raidz1-0                ONLINE
            13283347160590042564  UNAVAIL  corrupted data
            16981727992215676534  UNAVAIL  corrupted data
            6607570030658834339   UNAVAIL  corrupted data
            3435463242860701988   UNAVAIL  corrupted data

When we use glabel for the same purpose with the combination of gnop,
the pool imports fine.

Any suggestions?