[Bug 241980] panic: I/O to pool appears to be hung on vdev

Sun Nov 24 06:54:02 UTC 2019

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241980

--- Comment #21 from Eugene Grosbein <eugen at freebsd.org> ---
It took me unexpectedly long time to get debugging output as I struggled to
force the loader perform one-time (nextboot) loading for patched zfs.ko being
able to perform only one reboot per day for this production machine. It
appeared we have documented way to do this for the kernel but not for modules.
After several attempts, zfs_name="/boot/nextboot/zfs.ko" in the
/boot/nextboot.conf did it finally.

# TZ=UTC sysctl kern.boottime
kern.boottime: { sec = 1574393119, usec = 301171 } Fri Nov 22 03:25:19 2019

Local time is UTC+3. After two days of uptime, I got this in the log:

Nov 24 06:24:36 col02 kernel: sata SLOW IO: zio io_type 3 timestamp
171750435607373ns, delta 1006734932093ns, last io 172757121535759ns
I/O to pool 'sata' appears to be hung on vdev guid 3313589389580178043 at
'/dev/da2.eli' active zio 65

I've used this script to convert ZFS timestamps (getnanouptime expressed in
nanoseconds) to readable time:

#!/bin/sh
date -jr $(sysctl -n kern.boottime | awk -vt=${1%?????????} -F'[, ]' '{print
t+$4}')
#EOF

First, there was no hardware hung. The system is alive, both ZFS pools still
work, dd if=/dev/da2.eli works too. Second, ip_type 3 is ZIO_TYPE_FREE from
/sys/cddl/contrib/opensolaris/uts/common/sys/fs/zfs.h.

# gethrtime 172757121535759
Sun Nov 24 06:24:36 MSK 2019

65 active zio's correspond to L(q) shown by gstat:

# gstat -adI1s -f 'da[2-6].*'
dT: 1.001s  w: 1.000s  filter: da[2-6].*
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d  
%busy Name
   65    774     37   3924    9.0    540  25707    0.7    193   8532  212.6  
84.5| da2
   65    780     39   4144   14.3    544  26007    0.9    193   7724  241.7  
91.0| da3
   65    771     33   3345    9.3    542  24448    0.7    192   8100  255.5  
90.7| da4
   65    851     38   4004    7.8    554  28840    1.4    255   9679  236.0  
97.1| da5
   65    766     36   4100    4.5    534  22730    0.5    192   8911  208.8  
75.6| da6
   65    693     23   4144   14.8    474  26007    1.2    193   7724  243.0  
95.0| da3.eli
   65    691     20   3345    9.4    476  24448    1.0    192   8100  255.8  
95.1| da4.eli
   65    688     22   3924    9.8    470  25707    1.0    193   8532  213.0  
89.1| da2.eli
   65    753     22   4004   11.6    473  28840    1.7    255   9679  238.3 
100.7| da5.eli
   65    691     20   4100    5.6    476  22730    0.7    192   8911  210.6  
80.7| da6.eli

-- 
You are receiving this mail because:
You are the assignee for the bug.