Re: any way to recover from I/O hang?

On Fri, Nov 28, 2025 at 1:48 AM Anthony Pankov <anthony.pankov@yahoo.com> wrote:
>
>
> Friday, November 28, 2025, 1:06:19 AM, you wrote:
>
> > Without a wchan waiter or tracebacks, I can't help at all. Nothing you've
> > said is actionable, alas. umount should clear everything up, but if you've
> > hit some bug where the I/O never completes due to deadlock, nothing can
> > help you... except maybe tracebacks to we can understand what that deadlock
> > might be.
>
> I wander do kernel care about infinite I/O waiting? Neither the problem nor previous situation with ggate doesn't seem like a deadlock for me.
> If I remember correctly with ggate reviving ggate server do revive 'hanged' client which also was unresponsible to -KILL and 'umount -f'.
>
> Anyway I think it's very desirable that  'umount -f'  will apply more force and not give up with "device busy" message.
>
> I partially understand that "more force" is a complex problem. May be of something like replacing mountpoint vnode structures by deadfs structures.
This is not easy to do (as in unlikely that anyone will ever
implement it). The big issue is "if in progress file systems
are aborted, data is lost and the file system structure gets
messed up".  Even if people are willing to live with data
loss, there might be new cases that fsck(8) must handle.

The exercise of implementing this could be a major project,
since all sorts of msleep()s must be changed to return and
then all the code that calls them must handle that special case
without problems.  There is also be lots of waiters on locks
and buffer cache blocks that would need to be dealt with.

I did "umount -N" for the NFS client, which does what you
are asking for and it took quite a while to implement (and
I wouldn't be surprised if there is still bugs that make it
fail in rare cases).  NFS has the advantage that it does
not deal with file system structures like inodes and
indirect blocks, but it still wasn't easy to get all the
"waiters on buffer cache buffers" to error out, etc.

As Warner said "If you collect info during the hang and
post that, someone might be able to diagnose the
problem".
As a starting point, the output of "ps axHl" when the
hang occurs should show what the various processes
are waiting on. Once you have the "wchan" for the
processes/threads that are stuck, you can grep around
in the sources to find out what it is sleeping on.

It might also be a resource exhaustion situation, such
as "out of vnodes". I know that, when kern.maxvnodes is
exceeded, the system doesn't hang, but it can get so slow
that it appears hung. (That one is recognized by threads
with a wchan that starts with "v", but I cannot remember
the exact wchan.)

procstat -kk can also be useful, to figure out what locks
various processes are waiting for.

rick

>
>
> > On Thu, Nov 27, 2025 at 3:13 AM Anthony Pankov <anthony.pankov@yahoo.com>
> > wrote:
>
> >>
> >> Thursday, November 27, 2025, 7:58:21 AM, you wrote:
> >>
> >> > On Wed, Nov 26, 2025 at 7:16 AM Anthony Pankov <anthony.pankov@yahoo.com
> >> >
> >> > wrote:
> >>
> >> >>
> >> >> Recently I'm again facing a situation where some UFS (?) error on one
> >> >> partition forced me to reboot whole server and interrupt people's work.
> >> >> As I understand the brief is:
> >> >>
> >> >> Reason: A process in state 'D' (uninterruptible wait/sleep) is blocked
> >> on
> >> >> an I/O operation (e.g., waiting for a network file share, a disk
> >> operation,
> >> >> or a device that is no longer responding) at the kernel level. The
> >> process
> >> >> is not awaken to handle signals until the condition it is waiting for is
> >> >> resolved.
> >> >> Solution: You cannot kill a 'D' state process with any signal, including
> >> >> SIGKILL (kill -9). The condition must be resolved (e.g., fixing the
> >> network
> >> >> connection, waiting for the I/O to time out, or potentially unmounting
> >> the
> >> >> filesystem with umount -f if it's a mounted network share). In some rare
> >> >> cases involving buggy kernel code or hardware failure, a reboot may be
> >> the
> >> >> only option.
> >> >>
> >> >> In my case there was a jailed samba which share one ufs-formatted
> >> >> partition. The samba processes hangs one by one and sharing has stopped
> >> it
> >> >> work.
> >> >> There was no message in log. Geom reported no errors. Disk's has no
> >> error.
> >> >> It seems that there was some introduced inconsistency in UFS.
> >> >> My thoughts was to stop/kill samba, remove samba's jail, unmount
> >> partition
> >> >> and do fsck.
> >> >>
> >> >> But 'jail -r' exit with something 'rc.shutdown exited ... 9' and left
> >> jail
> >> >> running. pskill -KILL for samba processes say nothing and do nothing.
> >> >> 'umount -f' say 'Device is busy'. Various utilities such as 'df' hangs.
> >> So
> >> >> I forced to shutdown the server which has other important and workable
> >> >> service to resolve the situation.
> >> >>
> >> >> I wander is there any way to treat such a cases? May be 'umount -f' can
> >> >> have more power...
> >> >>
> >>
> >> > So no errors from geom? Or from CAM? what's the underlying storage
> >> > hardware? And what's the wchan / straceback for the processes in 'D'
> >> state?
> >> > And do you know the location of the file that's waiting for I/O?
> >>
> >> No errors. UFS was on gmirror and 'gmirror status' said 'OK'.
> >> Unfortunately I was unable to do meaningful investigation under stress.
> >> There was a number of smbd process in a jail which ignore killing and
> >> anything else.
> >> The first inconsistency found by fsck later was  a file with mtime
> >> corresponding to a first claim of a problem from one user. Hour later there
> >> was more claims and finally samba share stop working.  As I understand UFS
> >> inconsistency grew over that hour. Because there was multiple smbd
> >> processes each corresponding to different user there it is no one file
> >> maked problem.
> >>
> >>
> >> > There's a number of things that might cause this, but they typically are
> >> > noisy about it. There's a couple of deadlock issues that might cause
> >> this,
> >> > but getting some more information is needed to understand what might be
> >> > done to mitigate or prevent the deadlocks.
> >>
> >> > P.S. Previous situation was when I do simple (as I think) experiments
> >> with
> >> >> ggated and have forced to reboot server when 'mount' hung.
> >> >>
> >>
> >> > I assume ggate isn't involved when this happens now, right?
> >>
> >> That's right.  I note this because it seems that stoping ggate server
> >> while running ggate client is the simplest way to reproduce the problem.
> >>
> >> P.S. I was really hope that "umount -f" awake waiting processes for
> >> killing.
> >>
> >>
> >>
> >>
> >> >> --
> >> >> Best regards,
> >> >>  Anthony Pankov                         mailto:anthony.pankov@yahoo.com
> >> >>
> >> >>
> >> >>
> >>
> >>
> >> --
> >> Best regards,
> >> Anthony
> >>
> >>
>
>
> --
> Best regards,
> Anthony
>
>