Booting off ZFS pool with failed ZIL/cache device

[Alan Somers]

On Sun, Jul 29, 2018 at 4:43 AM, Stefan Bethke <stb at lassitu.de> wrote:

> Folks,
>
> my ZIL/cache SSD apparently just died. Rebooting the system with the SATA
> M.2 SSD hung, so I removed the card from the system.
>
> On powerup, loader acts normally, all four SATA disks (main Raid-Z1
> devices) are all probed successfully, but mount root fails:
>
> ZFS filesystem version: 5
> ZFS storage pool version: features support (5000)
> Timecounters tick every 1.000 msec
> ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
> ada0: <TOSHIBA MD04ACA50D FS2A> ATA8-ACS SATA 3.x device
> ada0: Serial Number Z5Q7K0RIFFRC
> ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
> ada0: Command Queueing enabled
> ada0: 4769307MB (9767541168 512 byte sectors)
> ada1 at ahcich2 bus 0 scbus2 target 0 lun 0
> ada1: <TOSHIBA MD04ACA50D FS2A> ATA8-ACS SATA 3.x device
> ada1: Serial Number Y5PIK0A2FFRC
> ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
> ada1: Command Queueing enabled
> ada1: 4769307MB (9767541168 512 byte sectors)
> ada2 at ahcich3 bus 0 scbus3 target 0 lun 0
> ada2: <TOSHIBA MD04ACA50D FS2A> ATA8-ACS SATA 3.x device
> ada2: Serial Number 36D2K0VZFFRC
> ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
> ada2: Command Queueing enabled
> ada2: 4769307MB (9767541168 512 byte sectors)
> ada3 at ahcich4 bus 0 scbus4 target 0 lun 0
> ada3: <TOSHIBA MD04ACA50D FS2A> ATA8-ACS SATA 3.x device
> ada3: Serial Number Z5SDK0J3FFRC
> ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
> ada3: Command Queueing enabled
> ada3: 4769307MB (9767541168 512 byte sectors)
> pass4 at ahciem0 bus 0 scbus6 target 0 lun 0
> pass4: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
> Trying to mount root from zfs:p2/be/11 []...
> GEOM_MIRROR: Device mirror/p2swap launched (4/4).
> random: unblocking device.
> Mounting from zfs:p2/be/11 failed with error 6; retrying for 3 more seconds
> Mounting from zfs:p2/be/11 failed with error 6.
>
> Loader variables:
>  vfs.root.mountfrom=zfs:p2/be/11
>
> Manual root filesystem specification:
>  <fstype>:<device> [options]
>      Mount <device> using filesystem <fstype>
>      and with the specified (optional) option list.
>
>    eg. ufs:/dev/da0s1a
>        zfs:tank
>        cd9660:/dev/cd0 ro
>          (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)
>
>  ?               List valid disk boot devices
>  .               Yield 1 second (for background tasks)
>  <empty line>    Abort manual input
>
> mountroot>
>
> Is there an easy way to boot into single user mode and remove the
> ZIL/cache devices that are not there anymore? Or do I need a USB key to
> boot off of and zfs import the pool first?
>
>
> Thanks,
> Stefan
>
> --
> Stefan Bethke <stb at lassitu.de>   Fon +49 151 14070811


A missing L2ARC is no problem, but a missing ZIL may mean filesystem
corruption.  Hopefully the ZIL was empty when it failed.  I would try
booting from ZFS and removing the offending device.  Good luck.
-Alan