Re: any way to recover from I/O hang?

Without a wchan waiter or tracebacks, I can't help at all. Nothing you've
said is actionable, alas. umount should clear everything up, but if you've
hit some bug where the I/O never completes due to deadlock, nothing can
help you... except maybe tracebacks to we can understand what that deadlock
might be.

Warner

On Thu, Nov 27, 2025 at 3:13 AM Anthony Pankov <anthony.pankov@yahoo.com>
wrote:

>
> Thursday, November 27, 2025, 7:58:21 AM, you wrote:
>
> > On Wed, Nov 26, 2025 at 7:16 AM Anthony Pankov <anthony.pankov@yahoo.com
> >
> > wrote:
>
> >>
> >> Recently I'm again facing a situation where some UFS (?) error on one
> >> partition forced me to reboot whole server and interrupt people's work.
> >> As I understand the brief is:
> >>
> >> Reason: A process in state 'D' (uninterruptible wait/sleep) is blocked
> on
> >> an I/O operation (e.g., waiting for a network file share, a disk
> operation,
> >> or a device that is no longer responding) at the kernel level. The
> process
> >> is not awaken to handle signals until the condition it is waiting for is
> >> resolved.
> >> Solution: You cannot kill a 'D' state process with any signal, including
> >> SIGKILL (kill -9). The condition must be resolved (e.g., fixing the
> network
> >> connection, waiting for the I/O to time out, or potentially unmounting
> the
> >> filesystem with umount -f if it's a mounted network share). In some rare
> >> cases involving buggy kernel code or hardware failure, a reboot may be
> the
> >> only option.
> >>
> >> In my case there was a jailed samba which share one ufs-formatted
> >> partition. The samba processes hangs one by one and sharing has stopped
> it
> >> work.
> >> There was no message in log. Geom reported no errors. Disk's has no
> error.
> >> It seems that there was some introduced inconsistency in UFS.
> >> My thoughts was to stop/kill samba, remove samba's jail, unmount
> partition
> >> and do fsck.
> >>
> >> But 'jail -r' exit with something 'rc.shutdown exited ... 9' and left
> jail
> >> running. pskill -KILL for samba processes say nothing and do nothing.
> >> 'umount -f' say 'Device is busy'. Various utilities such as 'df' hangs.
> So
> >> I forced to shutdown the server which has other important and workable
> >> service to resolve the situation.
> >>
> >> I wander is there any way to treat such a cases? May be 'umount -f' can
> >> have more power...
> >>
>
> > So no errors from geom? Or from CAM? what's the underlying storage
> > hardware? And what's the wchan / straceback for the processes in 'D'
> state?
> > And do you know the location of the file that's waiting for I/O?
>
> No errors. UFS was on gmirror and 'gmirror status' said 'OK'.
> Unfortunately I was unable to do meaningful investigation under stress.
> There was a number of smbd process in a jail which ignore killing and
> anything else.
> The first inconsistency found by fsck later was  a file with mtime
> corresponding to a first claim of a problem from one user. Hour later there
> was more claims and finally samba share stop working.  As I understand UFS
> inconsistency grew over that hour. Because there was multiple smbd
> processes each corresponding to different user there it is no one file
> maked problem.
>
>
> > There's a number of things that might cause this, but they typically are
> > noisy about it. There's a couple of deadlock issues that might cause
> this,
> > but getting some more information is needed to understand what might be
> > done to mitigate or prevent the deadlocks.
>
> > P.S. Previous situation was when I do simple (as I think) experiments
> with
> >> ggated and have forced to reboot server when 'mount' hung.
> >>
>
> > I assume ggate isn't involved when this happens now, right?
>
> That's right.  I note this because it seems that stoping ggate server
> while running ggate client is the simplest way to reproduce the problem.
>
> P.S. I was really hope that "umount -f" awake waiting processes for
> killing.
>
>
>
>
> >> --
> >> Best regards,
> >>  Anthony Pankov                         mailto:anthony.pankov@yahoo.com
> >>
> >>
> >>
>
>
> --
> Best regards,
> Anthony
>
>