ZFS With Gpart partitions
Dan Carroll
fbsd at dannysplace.net
Sun Jan 1 22:04:17 UTC 2012
On 1/01/2012 10:45 PM, Freddie Cash wrote:
> On Sun, Jan 1, 2012 at 4:31 AM, Dan Carroll<fbsd at dannysplace.net> wrote:
>>
>> Ok, so the label is still there. Two things are still strange.
>> 1) data2 is not in /dev/gpt/
>> 2) "glabel list da8p1" does not find anything. But it does for other
>> drives.
>
> You may need to offline the disk in the zfs pool, then "gpart destroy
> da8" to clear out the primary GPT table at the start of the disk, and
> the secondary GPT table at the end of the disk. That will also clear
> out the /dev/da8p* entries, and related /dev/gpt/ entry.
>
> Then create the GPT from scratch, including the label.
>
> Then, finally, add/replace the disk to the ZFS pool.
>
> The [kernel|GEOM|whatever] doesn't do a very good job of re-checking
> the disks for changed GPT tables, instead using the ones in memory.
> Or something along those lines. You have to destroy the existing
> table, then create it, which causes the [kernel|GEOM|whatever] to
> re-taste the disk and load the GPT off the disk, thus creating the
> /dev/gpt/ entry.
Thanks for your reply. It's quite informative. Reading a little more I
see that the label stuff is written to the end of the disk. Here is
what I think happened.
1) I inserted a disk with a label that already existed. The kernel
decided to simply remove the device entry from /dev/gpt (data2) and so
ZFS found it's data on the GPT partition (rather than the label that
points to the partition).
2) I wiped the new disk on a second system, but actually I didn't wipe
the GPT label at the end until I re-did the GPT partition.
3) Re-inserted new disk and resilvered. All good but ZFS now knows the
old disk via the GPT partition.
I'm still confused as to why "glabel list da8p1" does not work.
Perhaps all that command does is: scan the device for a label (it would
find data2) and then check what data2 is in /dev/gpt?
I guess another way might be to bring down the server, take the disk
out, re-do the GPT stuff on another machine, and then put it back.
That way ZFS may well simpy see it's data on the labeled device.
Oh wait, that wont create the /dev/gpt entries... :-(
I'd really like to fix this but avoid doing *another* resilver. The
first one is finished now and a scrub operation will be done in an hour
or so. I guess I should have the faith to do go ahead and do it again,
but there is an element of risk, and it's to fix something cosmetic.
That feels wrong...
>> Also, is there a difference (for ZFS) acessing the drive via /dev/da8p1 or
>> /dev/gpt/data2
>
> As far as ZFS is concerned, no. When importing the pool, ZFS checks
> for ZFS metadata to determine which disk belongs to which vdev in
> which pool, and gets all the pathing sorted out automatically.
>
> However, since disk device nodes can change at boot (add a new
> controller, boot with a dead drive, do a BIOS update that reverses the
> PCI scan order, etc), it makes it easier on the admin to have labels.
Yeah I am aware of ZFS doing that. It's a nice feature. It's also
the reason why in my system data2 is NOT da2p1.
What I was trying to dertermine is if there is any danger to ZFS
accessing the disk in this manner. Is da8p1 an identical device to data3?
From what you describe, from ZFS' point of view, for each device in the
pool, there are actually 2 versions of it. The dataX version, and the
daXp1 version..
I am guessing that ZFS keeps track of the disk names in it's pools.
Otherwise, what would stop this (the label being replaced with a device
name in ZFS) happening between reboots?
>
>
> For example, in our 16-bay storage servers, I've set up a coordinate
> system where columns are letters A-D, and rows are numbers 1-4. Then
> I've labelled each disk according to where it is in the chassis:
> /dev/gpt/disk-a1
> /dev/gpt/disk-a2
> /dev/gpt/disk-a3
> /dev/gpt/disk-a4
> etc
>
> That way, no matter how the drives are actually enumerated by the
> BIOS, loader, kernel, drivers, etc, I always know which disk is having
> issues. :)
>
That was also my reasoning. My numbering starts disk0 being in the top
left slot, and continues row by row.
-D
More information about the freebsd-fs
mailing list