Errors on a file on a zpool: How to remove?

[Rich]

> This is a non-redundant pool. The remove command will not work. Replace
> will, but for that pool to function at all, *every* device must be
> present. If the metadata was recoverable, I think that the scrub would
> have reported "xxx kb repaired".

Actually, zpool remove can't be used for that either - zpool detach
gets used for anything that's "live".

> From http://dlc.sun.com/osol/docs/content/ZFSADMIN/gbbwl.html:
>
>    If the object number to a file path cannot be successfully translated,
>    either due to an error or because the object doesn't have a real file
>    path associated with it , as is the case for a dnode_t, then the
>    dataset name followed by the object's number is displayed. For
>    example:
>
>     monkey/dnode:<0x0>
>
> Which seems to be precisely your error. Continuing:
>
>    Then, try removing the file with the rm command. If this command
>    doesn't work, the corruption is within the file's metadata, and ZFS
>    cannot determine which blocks belong to the file in order to remove
>    the corruption.
>
>    If the corruption is within a directory or a file's metadata, the only
>    choice is to move the file elsewhere. You can safely move any file or
>    directory to a less convenient location, allowing the original object
>    to be restored in place."
>
> In other words, either move the files out of the way or restore the pool.
> I'd wager that any other filesystem would have simply wiped out entire
> directory trees or possibly just panicked with this kind of corruption.

I'm presuming you mean "to another FS", since we already saw rm reports EIEIO.

I've started that process, and I'll get out the backups for all the other data.

Thanks to everyone for being so helpful, and especially to you, Wes,
for citing exactly what was going on with the manual. All hail
manuals! :)

- Rich