cache devices come up as dsk/original_device_name in zpools

[Attila Nagy]

On 10/14/11 11:00, Jeremy Chadwick wrote:
> On Fri, Oct 14, 2011 at 10:47:12AM +0200, Attila Nagy wrote:
>> Hi,
>>
>> A have a zpool with cache devices on 8-STABLE (csuped and compiled
>> at Sep 14 15:01:25 CEST 2011). The problem is every time I reboot,
>> the cache devices turn to UNAVAIL (because device name changes to
>> dsk/daXX):
>> dsk/da37 UNAVAIL 0 0 0 cannot open
>> dsk/da38 UNAVAIL 0 0 0 cannot open
>>
>> After removing and re-adding them, everyting goes back to normal,
>> until the next reboot. I have no /boot/zfs/zpool.cache (because the
>> machine is netbooted), maybe this is the cause? In previous versions
>> everything was fine.
> Obviously at some point when you built this system you entered
> "dsk/da37" and "dsk/da38".  So the metadata on those drives probably
> contains references to those strings.  You need to clear/change that.
Pretty unlikely. And given that this happens on all machines, upgraded 
to a recent 8-STABLE (and never happened before), I would say something 
has been changed regarding this.
In the user space tools there are a lot of occurrences of /dev/dsk...
>
> I'm not sure how to go about doing that, especially on a system which
> lacks /boot/zfs/zpool.cache.  A one-time "zpool export" then a reboot, I
> imagine, would suffice, but I'm not sure if export actually changes the
> metadata on the disk itself or just updates the zpool.cache file.
>
> If you ran "zdb" on this system (the output will be HUGE given the
> number of vdevs and devices you have!), you should see some relevant
> information under each disk (child), specifically "path" vs.
> "phys_path".  Maybe these differ?
Well, zdb seems to be quite useless without zpool.cache...
But I found a spare machine where I could do an export. The zdb output 
does not contain the above disks (da37, da38, the cache devices).
I let the tool run for about 30 minutes only.
>
> You might also try tinkering about with the loader.conf(5) variables
> zpool_cache_*.  Depending on your setup, you might be able to move the
> zpool.cache file to a different location -- I realise you PXE boot, but
> if you have any sort of storage media on that system that isn't under
> ZFS that *is* available (e.g. a small UFS partition, etc.) then you
> might consider storing it there.  See /boot/defaults/loader.conf.
I have no UFS on these machines, and this is just fine. And was fine 
always, I hope this stays this way. :)
>
> Otherwise I'm not sure how to go about changing the actual strings in
> the disk metadata.  Maybe remove the cache devices entirely, zero out
> the first and last ~16MBytes of the da37 and da38 disks (using dd), then
> re-add them using their "daXX" name?  That might suffice.
>
I'm not sure whether this is in the on-disk metadata. How could I add a 
/dev/dsk/da38 disk with zpool? It does not exist.