recovering from or increasing timeouts on virtio block device
Bryan Venteicher
bryanv at freebsd.org
Tue Feb 18 17:14:53 UTC 2014
On Tue, Feb 18, 2014 at 10:57 AM, John Nielsen <lists at jnielsen.net> wrote:
> On Feb 18, 2014, at 3:32 AM, Edward Tomasz Napierała <trasz at freebsd.org>
> wrote:
>
> > Wiadomość napisana przez John Nielsen w dniu 17 lut 2014, o godz. 21:21:
> >> I run several FreeBSD virtual machines in a Linux KVM environment with
> a SAN. The VMs use virtio block storage, and the KVM hosts map the virtual
> volumes to targets on the SAN. Occasionally, failover or other maintenance
> events on the SAN cause it to be unavailable for 30+ seconds. When this
> happens, the FreeBSD VMs have hard failures on the vtbd* devices, and
> thereafter any attempted reads or writes return immediately with an error
> (even after the SAN is responsive again). The only way to recover a VM once
> that happens is to hard boot it.
> >>
> >> Is there any way to adjust the timeouts or enable some kind of retry
> for the virtio block devices? It would be nice to be able to recover
> gracefully after a SAN event without needing to reboot the VMs.
> >
> > Use gmountver(8) perhaps?
>
> Thanks for the tip (and for writing it :), I haven't encountered that one
> before. I will experiment with it but I'm not sure it's a fit for this
> particular scenario (at least not by itself). When a SAN event happens the
> virtual machine's vtbd0 device doesn't disappear, the underlying hardware
> just fails to respond for a long-ish time. I suspect that the driver gives
> up after either a certain length of time or number of errors, but my C
> driver-fu isn't up to figuring it out exactly. Once it gives up, any I/O
> requests to the (still "present") device fail immediately, and I can't see
> a way to get the driver to actually try any (new or old) I/O again.
>
The vtbd driver has no internal retry mechanism, and pays no attention to
errors other than report then, and never gives up :)
It is not clear to me whether IO is getting turned around in FreeBSD before
it reaches the driver, or within the host. Do you continue to see "hard
error ..." messages on the console?
>
> JN
>
>
More information about the freebsd-stable
mailing list