zfs on nvme: gnop breaks pool, zfs gets stuck
Karli Sjöberg
karli.sjoberg at slu.se
Wed Apr 27 16:42:10 UTC 2016
Den 27 apr 2016 15:23 skrev Gerrit Kühn <gerrit.kuehn at aei.mpg.de>:
>
> Hello all,
>
> I have a set of three NVME-ssds on PCIe-converters:
>
> ---
> root at storage:~ # nvmecontrol devlist
> nvme0: SAMSUNG MZVPV512HDGL-00000
> nvme0ns1 (488386MB)
> nvme1: SAMSUNG MZVPV512HDGL-00000
> nvme1ns1 (488386MB)
> nvme2: SAMSUNG MZVPV512HDGL-00000
> nvme2ns1 (488386MB)
> ---
>
>
> I want to use a z1 raid on these and created 1m-aligned partitions:
>
> ---
> root at storage:~ # gpart show
> => 34 1000215149 nvd0 GPT (477G)
> 34 2014 - free - (1.0M)
> 2048 1000212480 1 freebsd-zfs (477G)
> 1000214528 655 - free - (328K)
>
> => 34 1000215149 nvd1 GPT (477G)
> 34 2014 - free - (1.0M)
> 2048 1000212480 1 freebsd-zfs (477G)
> 1000214528 655 - free - (328K)
>
> => 34 1000215149 nvd2 GPT (477G)
> 34 2014 - free - (1.0M)
> 2048 1000212480 1 freebsd-zfs (477G)
> 1000214528 655 - free - (328K)
> ---
>
>
> After creating a zpool I recognized that it was using ashift=9. I vaguely
> remembered that SSDs usually have 4k (or even larger) sectors, so I
> destroyed the pool and set up gnop-providers with -S 4k to get ashift=12.
> This worked as expected:
>
> ---
> pool: flash
> state: ONLINE
> scan: none requested
> config:
>
> NAME STATE READ WRITE CKSUM
> flash ONLINE 0 0 0
> raidz1-0 ONLINE 0 0 0
> gpt/flash0.nop ONLINE 0 0 0
> gpt/flash1.nop ONLINE 0 0 0
> gpt/flash2.nop ONLINE 0 0 0
>
> errors: No known data errors
> ---
>
>
> This pool can be used, exported and imported just fine as far as I can
> tell. Then I exported the pool and destroyed the gnop-providers. When
> starting with "advanced format" hdds some years ago, this was the way to
> make zfs recognize the disks with ashift=12. However, destroying the
> gnop-devices appears to have crashed the pool in this case:
>
> ---
> root at storage:~ # zpool import
> pool: flash
> id: 4978839938025863522
> state: ONLINE
> status: One or more devices contains corrupted data.
> action: The pool can be imported using its name or numeric identifier.
> see: http://illumos.org/msg/ZFS-8000-4J
> config:
>
> flash ONLINE
> raidz1-0 ONLINE
> 11456367280316708003 UNAVAIL corrupted
> data gptid/55ae71aa-eb84-11e5-9298-0cc47a6c7484 ONLINE
> 6761786983139564172 UNAVAIL corrupted
> data
> ---
>
>
> How can the pool be online, when two of three devices are unavailable? I
> tried to import the pool nevertheless, but the zpool command got stuck in
> state tx-tx. "soft" reboot got stuck, too. I had to push the reset button
> to get my system back (still with a corrupt pool). I cleared the labels
> and re-did everything: the issue is perfectly reproducible.
>
> Am I doing something utterly wrong? Why is removing the gnop-nodes
> tampering with the devices (I think I did exactly this dozens of times on
> normal hdds during that previous years, and it always worked just fine)?
> And finally, why does the zpool import fail without any error message and
> requires me to reset the system?
> The system is 10.2-RELEASE-p9, update is scheduled for later this week
> (just in case it would make sense to try this again with 10.3). Any other
> hints are most welcome.
>
>
> cu
> Gerrit
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
You probably need to tell zfs "where" the providers are:
# zpool import -d /dev/gpt flash
/K
More information about the freebsd-fs
mailing list