Re: any way to recover from I/O hang?

Friday, November 28, 2025, 1:06:19 AM, you wrote:

> Without a wchan waiter or tracebacks, I can't help at all. Nothing you've
> said is actionable, alas. umount should clear everything up, but if you've
> hit some bug where the I/O never completes due to deadlock, nothing can
> help you... except maybe tracebacks to we can understand what that deadlock
> might be.

I wander do kernel care about infinite I/O waiting? Neither the problem nor previous situation with ggate doesn't seem like a deadlock for me.
If I remember correctly with ggate reviving ggate server do revive 'hanged' client which also was unresponsible to -KILL and 'umount -f'. 

Anyway I think it's very desirable that  'umount -f'  will apply more force and not give up with "device busy" message.

I partially understand that "more force" is a complex problem. May be of something like replacing mountpoint vnode structures by deadfs structures.


> On Thu, Nov 27, 2025 at 3:13 AM Anthony Pankov <anthony.pankov@yahoo.com>
> wrote:

>>
>> Thursday, November 27, 2025, 7:58:21 AM, you wrote:
>>
>> > On Wed, Nov 26, 2025 at 7:16 AM Anthony Pankov <anthony.pankov@yahoo.com
>> >
>> > wrote:
>>
>> >>
>> >> Recently I'm again facing a situation where some UFS (?) error on one
>> >> partition forced me to reboot whole server and interrupt people's work.
>> >> As I understand the brief is:
>> >>
>> >> Reason: A process in state 'D' (uninterruptible wait/sleep) is blocked
>> on
>> >> an I/O operation (e.g., waiting for a network file share, a disk
>> operation,
>> >> or a device that is no longer responding) at the kernel level. The
>> process
>> >> is not awaken to handle signals until the condition it is waiting for is
>> >> resolved.
>> >> Solution: You cannot kill a 'D' state process with any signal, including
>> >> SIGKILL (kill -9). The condition must be resolved (e.g., fixing the
>> network
>> >> connection, waiting for the I/O to time out, or potentially unmounting
>> the
>> >> filesystem with umount -f if it's a mounted network share). In some rare
>> >> cases involving buggy kernel code or hardware failure, a reboot may be
>> the
>> >> only option.
>> >>
>> >> In my case there was a jailed samba which share one ufs-formatted
>> >> partition. The samba processes hangs one by one and sharing has stopped
>> it
>> >> work.
>> >> There was no message in log. Geom reported no errors. Disk's has no
>> error.
>> >> It seems that there was some introduced inconsistency in UFS.
>> >> My thoughts was to stop/kill samba, remove samba's jail, unmount
>> partition
>> >> and do fsck.
>> >>
>> >> But 'jail -r' exit with something 'rc.shutdown exited ... 9' and left
>> jail
>> >> running. pskill -KILL for samba processes say nothing and do nothing.
>> >> 'umount -f' say 'Device is busy'. Various utilities such as 'df' hangs.
>> So
>> >> I forced to shutdown the server which has other important and workable
>> >> service to resolve the situation.
>> >>
>> >> I wander is there any way to treat such a cases? May be 'umount -f' can
>> >> have more power...
>> >>
>>
>> > So no errors from geom? Or from CAM? what's the underlying storage
>> > hardware? And what's the wchan / straceback for the processes in 'D'
>> state?
>> > And do you know the location of the file that's waiting for I/O?
>>
>> No errors. UFS was on gmirror and 'gmirror status' said 'OK'.
>> Unfortunately I was unable to do meaningful investigation under stress.
>> There was a number of smbd process in a jail which ignore killing and
>> anything else.
>> The first inconsistency found by fsck later was  a file with mtime
>> corresponding to a first claim of a problem from one user. Hour later there
>> was more claims and finally samba share stop working.  As I understand UFS
>> inconsistency grew over that hour. Because there was multiple smbd
>> processes each corresponding to different user there it is no one file
>> maked problem.
>>
>>
>> > There's a number of things that might cause this, but they typically are
>> > noisy about it. There's a couple of deadlock issues that might cause
>> this,
>> > but getting some more information is needed to understand what might be
>> > done to mitigate or prevent the deadlocks.
>>
>> > P.S. Previous situation was when I do simple (as I think) experiments
>> with
>> >> ggated and have forced to reboot server when 'mount' hung.
>> >>
>>
>> > I assume ggate isn't involved when this happens now, right?
>>
>> That's right.  I note this because it seems that stoping ggate server
>> while running ggate client is the simplest way to reproduce the problem.
>>
>> P.S. I was really hope that "umount -f" awake waiting processes for
>> killing.
>>
>>
>>
>>
>> >> --
>> >> Best regards,
>> >>  Anthony Pankov                         mailto:anthony.pankov@yahoo.com
>> >>
>> >>
>> >>
>>
>>
>> --
>> Best regards,
>> Anthony
>>
>>


-- 
Best regards,
Anthony