Re: any way to recover from I/O hang?

On Fri, Nov 28, 2025 at 2:48 AM Anthony Pankov <anthony.pankov@yahoo.com>
wrote:

>
> Friday, November 28, 2025, 1:06:19 AM, you wrote:
>
> > Without a wchan waiter or tracebacks, I can't help at all. Nothing you've
> > said is actionable, alas. umount should clear everything up, but if
> you've
> > hit some bug where the I/O never completes due to deadlock, nothing can
> > help you... except maybe tracebacks to we can understand what that
> deadlock
> > might be.
>
> I wander do kernel care about infinite I/O waiting?


Yes. We try really hard never to wait forever. Waiting forever is a bug.
However, there's no way to solve it generically, though. There's no common
choke point to use, nor any way to really communicate the underlying system
is gone. It all depends on the exact specifics what's wrong and how to fix
it.


> Neither the problem nor previous situation with ggate doesn't seem like a
> deadlock for me.
>

They are. Whether they seem to you or not, failing to wakeup sleepers is by
definition a deadlock. It might not be, exactly, two locked processes
waiting for another process to be able to make progress, but it is
something that's leaving dangling resources that it didn't clean up and if
you don't know why, you can't prevent it in the future.


> If I remember correctly with ggate reviving ggate server do revive
> 'hanged' client which also was unresponsible to -KILL and 'umount -f'.
>

I can't debug rumors. I hate to be harsh, but this isn't helpful. It's too
vague to be actionable or to explain exactly what happened to bring that
situation about at a micro-level.


> Anyway I think it's very desirable that  'umount -f'  will apply more
> force and not give up with "device busy" message.
>

Yes. I believe in motherhood and apple pie too. I'm wanting to do something
to fix any issues we still have here.


> I partially understand that "more force" is a complex problem. May be of
> something like replacing mountpoint vnode structures by deadfs structures.


Without knowing what the problem is, it's impossible to know if this would
work or not. We have to know the specifics in order to know what's going
on. We already have all that stuff in place, but if something is sleeping
and nobody ever wakes it up, there's nothing to be done except fix that
bug. Tossing vague notions about what should be done isn't helpful.

Next time this happens, please please please panic the system, take a core
dump and contact me so I can get a kernel, the debug symbols and the crash
dump to look at. It would be even better if sources / source hash was
available.

Warner


>

> On Thu, Nov 27, 2025 at 3:13 AM Anthony Pankov <anthony.pankov@yahoo.com>
> > wrote:
>
> >>
> >> Thursday, November 27, 2025, 7:58:21 AM, you wrote:
> >>
> >> > On Wed, Nov 26, 2025 at 7:16 AM Anthony Pankov <
> anthony.pankov@yahoo.com
> >> >
> >> > wrote:
> >>
> >> >>
> >> >> Recently I'm again facing a situation where some UFS (?) error on one
> >> >> partition forced me to reboot whole server and interrupt people's
> work.
> >> >> As I understand the brief is:
> >> >>
> >> >> Reason: A process in state 'D' (uninterruptible wait/sleep) is
> blocked
> >> on
> >> >> an I/O operation (e.g., waiting for a network file share, a disk
> >> operation,
> >> >> or a device that is no longer responding) at the kernel level. The
> >> process
> >> >> is not awaken to handle signals until the condition it is waiting
> for is
> >> >> resolved.
> >> >> Solution: You cannot kill a 'D' state process with any signal,
> including
> >> >> SIGKILL (kill -9). The condition must be resolved (e.g., fixing the
> >> network
> >> >> connection, waiting for the I/O to time out, or potentially
> unmounting
> >> the
> >> >> filesystem with umount -f if it's a mounted network share). In some
> rare
> >> >> cases involving buggy kernel code or hardware failure, a reboot may
> be
> >> the
> >> >> only option.
> >> >>
> >> >> In my case there was a jailed samba which share one ufs-formatted
> >> >> partition. The samba processes hangs one by one and sharing has
> stopped
> >> it
> >> >> work.
> >> >> There was no message in log. Geom reported no errors. Disk's has no
> >> error.
> >> >> It seems that there was some introduced inconsistency in UFS.
> >> >> My thoughts was to stop/kill samba, remove samba's jail, unmount
> >> partition
> >> >> and do fsck.
> >> >>
> >> >> But 'jail -r' exit with something 'rc.shutdown exited ... 9' and left
> >> jail
> >> >> running. pskill -KILL for samba processes say nothing and do nothing.
> >> >> 'umount -f' say 'Device is busy'. Various utilities such as 'df'
> hangs.
> >> So
> >> >> I forced to shutdown the server which has other important and
> workable
> >> >> service to resolve the situation.
> >> >>
> >> >> I wander is there any way to treat such a cases? May be 'umount -f'
> can
> >> >> have more power...
> >> >>
> >>
> >> > So no errors from geom? Or from CAM? what's the underlying storage
> >> > hardware? And what's the wchan / straceback for the processes in 'D'
> >> state?
> >> > And do you know the location of the file that's waiting for I/O?
> >>
> >> No errors. UFS was on gmirror and 'gmirror status' said 'OK'.
> >> Unfortunately I was unable to do meaningful investigation under stress.
> >> There was a number of smbd process in a jail which ignore killing and
> >> anything else.
> >> The first inconsistency found by fsck later was  a file with mtime
> >> corresponding to a first claim of a problem from one user. Hour later
> there
> >> was more claims and finally samba share stop working.  As I understand
> UFS
> >> inconsistency grew over that hour. Because there was multiple smbd
> >> processes each corresponding to different user there it is no one file
> >> maked problem.
> >>
> >>
> >> > There's a number of things that might cause this, but they typically
> are
> >> > noisy about it. There's a couple of deadlock issues that might cause
> >> this,
> >> > but getting some more information is needed to understand what might
> be
> >> > done to mitigate or prevent the deadlocks.
> >>
> >> > P.S. Previous situation was when I do simple (as I think) experiments
> >> with
> >> >> ggated and have forced to reboot server when 'mount' hung.
> >> >>
> >>
> >> > I assume ggate isn't involved when this happens now, right?
> >>
> >> That's right.  I note this because it seems that stoping ggate server
> >> while running ggate client is the simplest way to reproduce the problem.
> >>
> >> P.S. I was really hope that "umount -f" awake waiting processes for
> >> killing.
> >>
> >>
> >>
> >>
> >> >> --
> >> >> Best regards,
> >> >>  Anthony Pankov                         mailto:
> anthony.pankov@yahoo.com
> >> >>
> >> >>
> >> >>
> >>
> >>
> >> --
> >> Best regards,
> >> Anthony
> >>
> >>
>
>
> --
> Best regards,
> Anthony
>
>