LSI 9240-4i 4K alignment
George Kontostanos
gkontos.mail at gmail.com
Sun Aug 19 18:07:15 UTC 2012
On Sun, Aug 19, 2012 at 8:58 PM, Don Lewis <truckman at freebsd.org> wrote:
> On 8 Aug, George Kontostanos wrote:
>> Hi all,
>>
>> We have a server with a LSI 9240-4i controller configured in JBOD with
>> 4 SATA disks. Running FreeBSD 9.1-Beta1:
>>
>> Relevant dmesg:
>>
>> FreeBSD 9.1-BETA1 #0: Thu Jul 12 09:38:51 UTC 2012
>> root at farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
>> CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3200.09-MHz K8-class CPU)
>> Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7
>> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>> Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
>> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
>> AMD Features2=0x1<LAHF>
>> TSC: P-state invariant, performance statistics
>> real memory = 17179869184 (16384 MB)
>> avail memory = 16471670784 (15708 MB)
>> ...
>> mfi0: <Drake Skinny> port 0xe000-0xe0ff mem
>> 0xf7a60000-0xf7a63fff,0xf7a00000-0xf7a3ffff irq 16 at device 0.0 on
>> pci1
>> mfi0: Using MSI
>> mfi0: Megaraid SAS driver Ver 4.23
>> ...
>> mfi0: 321 (397672301s/0x0020/info) - Shutdown command received from host
>> mfi0: 322 (boot + 3s/0x0020/info) - Firmware initialization started
>> (PCI ID 0073/1000/9241/1000)
>> mfi0: 323 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664
>> mfi0: 324 (boot + 3s/0x0020/info) - Firmware initialization started
>> (PCI ID 0073/1000/9241/1000)
>> mfi0: 325 (boot + 3s/0x0020/info) - Firmware version 2.130.354-1664
>> mfi0: 326 (boot + 5s/0x0020/info) - Package version 20.10.1-0107
>> mfi0: 327 (boot + 5s/0x0020/info) - Board Revision 03A
>> mfi0: 328 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3)
>> ...
>> mfisyspd0 on mfi0
>> mfisyspd0: 1907729MB (3907029168 sectors) SYSPD volume
>> mfisyspd0: SYSPD volume attached
>> mfisyspd1 on mfi0
>> mfisyspd1: 1907729MB (3907029168 sectors) SYSPD volume
>> mfisyspd1: SYSPD volume attached
>> mfisyspd2 on mfi0
>> mfisyspd2: 1907729MB (3907029168 sectors) SYSPD volume
>> mfisyspd2: SYSPD volume attached
>> mfisyspd3 on mfi0
>> mfisyspd3: 1907729MB (3907029168 sectors) SYSPD volume
>> mfisyspd3: SYSPD volume attached
>> ...
>> mfi0: 329 (boot + 25s/0x0002/info) - Inserted: PD 04(e0xff/s3) Info:
>> enclPd=ffff, scsiType=0, portMap=00,
>> sasAddr=4433221100000000,0000000000000000
>> mfi0: 330 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1)
>> mfi0: 331 (boot + 25s/0x0002/info) - Inserted: PD 05(e0xff/s1) Info:
>> enclPd=ffff, scsiType=0, portMap=02,
>> sasAddr=4433221102000000,0000000000000000
>> mfi0: 332 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2)
>> mfi0: 333 (boot + 25s/0x0002/info) - Inserted: PD 06(e0xff/s2) Info:
>> enclPd=ffff, scsiType=0, portMap=03,
>> sasAddr=4433221101000000,0000000000000000
>> mfi0: 334 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0)
>> mfi0: 335 (boot + 25s/0x0002/info) - Inserted: PD 07(e0xff/s0) Info:
>> enclPd=ffff, scsiType=0, portMap=01,
>> sasAddr=4433221103000000,0000000000000000
>> mfi0: 336 (397672376s/0x0020/info) - Time established as 08/07/12
>> 16:32:56; (28 seconds since power on)
>>
>> The problem:
>>
>> When trying to create a RaidZ pool using gpart and perform a 4K
>> alignment using gnop, we get the follwoing error immediately after
>> exporting the pool and destroying the .nop devices:
>>
>> id: 8043746387654554958
>> state: FAULTED
>> status: One or more devices contains corrupted data.
>> action: The pool cannot be imported due to damaged devices or data.
>> The pool may be active on another system, but can be imported using
>> the '-f' flag.
>> see: http://illumos.org/msg/ZFS-8000-5E
>> config:
>>
>> Pool FAULTED corrupted data
>> raidz1-0 ONLINE
>> 13283347160590042564 UNAVAIL corrupted data
>> 16981727992215676534 UNAVAIL corrupted data
>> 6607570030658834339 UNAVAIL corrupted data
>> 3435463242860701988 UNAVAIL corrupted data
>
> I'm planning on doing something similar, but I'm curious about how gnop
> and GPT labels interact. I want to partition the drives for my pool
> slightly on the small side so that I'm less likely to run into problems
> if I have to replace a drive in the future. If I used gpart to create
> and label a GPT partition on the drive, the partition will show up as
> /dev/gpt/label. The gnop man page says that running gnop on dev creates
> /dev/dev.nop. What happens if you gnop /dev/gpt/label? Is this what
> you are doing?
>
>> When we use glabel for the same purpose with the combination of gnop,
>> the pool imports fine.
>>
>> Any suggestions?
>
> It should be sufficient to only gnop one of the devices. You should be
> able to create the pool with only one gnop device to get the ashift
> value that you desire, export the pool, destroy the .nop device, and
> import the pool. If it things the device is corrupted (which seems like
> a bug of some sort), then ZFS should be able to resilver it.
>
After a few tests with many scenarios, the pool shows as ASHIFT=12
with the devices gnoped directly. However, the performance is
decreasing dramatically after adding a few TB of data. We haven't used
more than 30% of its capacity yet on a RaidZ1 pool.
Scrub speed has been reduced to 20M max. We will park the controller
for now and use the standard HBA supported by the mps driver :)
--
George Kontostanos
Aicom telecoms ltd
http://www.aisecure.net
More information about the freebsd-stable
mailing list