From nobody Fri Nov 28 13:22:16 2025 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dHv9N5Qd4z6HRH4 for ; Fri, 28 Nov 2025 13:22:32 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dHv9N3XGgz3kPY for ; Fri, 28 Nov 2025 13:22:32 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x531.google.com with SMTP id 4fb4d7f45d1cf-64320b9bb4bso3668028a12.0 for ; Fri, 28 Nov 2025 05:22:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1764336149; x=1764940949; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=clJf1v6sp9PRytcLp8EzTcj0QjjzsQacUIw7uvRDce4=; b=VT+gH5sgc6yDj1DHPgIOp7Heo4okk9J/F5GVce1pFWq9QkohShzeCaNsOo0WUke8Hb cPWKcv0Y6oKYQW2eN+KGq6A+HJm1S1X91bwfb6KbxiDatAWHSOcV91BmlkeIGPKklMCW r/M+uoNNh0xrQPkabsccTCClAW1uBqdfqDHdLF2tcy1CAksTMFI6KdqHg3RI2KeQZ54R Y6yeMIpDYpHqazp56WcgrefsrIES9BMc4FKiINY12oiQXDfgqQ/ezZ248RcecxYntP7r 3mXr5ueaXgzjECPbAayUpEYE3ADSayQseouIGeNJwjAKfs/pfruLILs62bTF1a/vc4LP 0XXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764336149; x=1764940949; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=clJf1v6sp9PRytcLp8EzTcj0QjjzsQacUIw7uvRDce4=; b=Tjmt2GeVdu2145Soc8Qe9gsrEArPR5y/54vwVh+jT3WkrTn8YPqBG/mPS1TR/17OXK D0X6g33pA8zcFd3VvDruMGPBXAdbPm1VLzHn5fOboB9EMTkBNdtEnxv+IPPd2JWOJnXH c8SgdIEVf63zAysTRKMUeqrJCYQTrPDc77EdHLL9Wt8y1pwor+yv0faow2BYibLrMirC oe04vHoCOzj/8TZf/RjAX18OITjArastUgU7yff2yZitu2fcQXgzoutSp6lE31wBoHPZ e1o1yPG96N6G/DWgpEX2TOspCWJhTHefLw4u00ZBU6Gk4wHUi5Q/nm3dxt4YQxI9e23X k3nw== X-Forwarded-Encrypted: i=1; AJvYcCVSlfl7pjDEamv4P7aTxbT+3ufzTVhaA3HDsfy6YTndwEnGS/wHvKhp1iZzUTQd6YUtmCiuPd32HzmyhOWZR74=@freebsd.org X-Gm-Message-State: AOJu0Yz00/sH7XZDdlQD/RnFhFqtNL5XL3jPSzoeokXz7vTr969PrZ3f l6WEF/fqxzDic8oi1wBZWg1GhM3E/5KQuahsg64umx8BzfI/akOZxvNQkdtpQe4Owzk6ETXqqkq DYsHCEBEqZW4p08iH5y9WqMjMz73rnQ== X-Gm-Gg: ASbGnctCT2C9WUNM7mucHGuQA0pw/WIikrduwnikMUbZMDcklNwvKaiSQYcnefBqnBL jlWLJ3oXnSyOis/IAVx5ogThEyM+Z61tp16DN4lvIas2uOkGk+9br30XwMqH0KaawiirxP9PsJ+ 9yveaiv9HZMd1QSs/DBjtTxtAcHMPYoWZ1JMlxEZTp53poSXCuLxlTXOAoqyzTxIngGsudknJG4 mRiKVT2ou832N3HpAXWm/uiZi668KVRbShq7P47K6IfQvhO1yVEWLzNwMikvefVVUvGA+uGcO5U Ggf3A9mfek8Bnx7oC3Y6oYmpcUVxxq4M4Y9a X-Google-Smtp-Source: AGHT+IFfAS5ZGrcKWa8XuCy0/b0ALpS/KWdj/9p5EzPtBKs7VTP6D1UeUleeLsi79uz23t7sQmr0eNgpEX4Kce8gpUc= X-Received: by 2002:a05:6402:35ca:b0:62e:ebb4:e6e0 with SMTP id 4fb4d7f45d1cf-64555078308mr24091278a12.1.1764336148921; Fri, 28 Nov 2025 05:22:28 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 References: <458400081.20251126171621.ref@yahoo.com> <458400081.20251126171621@yahoo.com> <908785764.20251127131317@yahoo.com> <1777092313.20251128124844@yahoo.com> In-Reply-To: <1777092313.20251128124844@yahoo.com> From: Rick Macklem Date: Fri, 28 Nov 2025 05:22:16 -0800 X-Gm-Features: AWmQ_bkAyq2kFL1gLqAbTotzumgk4cUh2PUtu0VUla1kOBOiMp8E7CjflOIy6zY Message-ID: Subject: Re: any way to recover from I/O hang? To: Anthony Pankov Cc: Warner Losh , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: 4dHv9N3XGgz3kPY On Fri, Nov 28, 2025 at 1:48=E2=80=AFAM Anthony Pankov wrote: > > > Friday, November 28, 2025, 1:06:19 AM, you wrote: > > > Without a wchan waiter or tracebacks, I can't help at all. Nothing you'= ve > > said is actionable, alas. umount should clear everything up, but if you= 've > > hit some bug where the I/O never completes due to deadlock, nothing can > > help you... except maybe tracebacks to we can understand what that dead= lock > > might be. > > I wander do kernel care about infinite I/O waiting? Neither the problem n= or previous situation with ggate doesn't seem like a deadlock for me. > If I remember correctly with ggate reviving ggate server do revive 'hange= d' client which also was unresponsible to -KILL and 'umount -f'. > > Anyway I think it's very desirable that 'umount -f' will apply more for= ce and not give up with "device busy" message. > > I partially understand that "more force" is a complex problem. May be of = something like replacing mountpoint vnode structures by deadfs structures. This is not easy to do (as in unlikely that anyone will ever implement it). The big issue is "if in progress file systems are aborted, data is lost and the file system structure gets messed up". Even if people are willing to live with data loss, there might be new cases that fsck(8) must handle. The exercise of implementing this could be a major project, since all sorts of msleep()s must be changed to return and then all the code that calls them must handle that special case without problems. There is also be lots of waiters on locks and buffer cache blocks that would need to be dealt with. I did "umount -N" for the NFS client, which does what you are asking for and it took quite a while to implement (and I wouldn't be surprised if there is still bugs that make it fail in rare cases). NFS has the advantage that it does not deal with file system structures like inodes and indirect blocks, but it still wasn't easy to get all the "waiters on buffer cache buffers" to error out, etc. As Warner said "If you collect info during the hang and post that, someone might be able to diagnose the problem". As a starting point, the output of "ps axHl" when the hang occurs should show what the various processes are waiting on. Once you have the "wchan" for the processes/threads that are stuck, you can grep around in the sources to find out what it is sleeping on. It might also be a resource exhaustion situation, such as "out of vnodes". I know that, when kern.maxvnodes is exceeded, the system doesn't hang, but it can get so slow that it appears hung. (That one is recognized by threads with a wchan that starts with "v", but I cannot remember the exact wchan.) procstat -kk can also be useful, to figure out what locks various processes are waiting for. rick > > > > On Thu, Nov 27, 2025 at 3:13=E2=80=AFAM Anthony Pankov > > wrote: > > >> > >> Thursday, November 27, 2025, 7:58:21 AM, you wrote: > >> > >> > On Wed, Nov 26, 2025 at 7:16=E2=80=AFAM Anthony Pankov >> > > >> > wrote: > >> > >> >> > >> >> Recently I'm again facing a situation where some UFS (?) error on o= ne > >> >> partition forced me to reboot whole server and interrupt people's w= ork. > >> >> As I understand the brief is: > >> >> > >> >> Reason: A process in state 'D' (uninterruptible wait/sleep) is bloc= ked > >> on > >> >> an I/O operation (e.g., waiting for a network file share, a disk > >> operation, > >> >> or a device that is no longer responding) at the kernel level. The > >> process > >> >> is not awaken to handle signals until the condition it is waiting f= or is > >> >> resolved. > >> >> Solution: You cannot kill a 'D' state process with any signal, incl= uding > >> >> SIGKILL (kill -9). The condition must be resolved (e.g., fixing the > >> network > >> >> connection, waiting for the I/O to time out, or potentially unmount= ing > >> the > >> >> filesystem with umount -f if it's a mounted network share). In some= rare > >> >> cases involving buggy kernel code or hardware failure, a reboot may= be > >> the > >> >> only option. > >> >> > >> >> In my case there was a jailed samba which share one ufs-formatted > >> >> partition. The samba processes hangs one by one and sharing has sto= pped > >> it > >> >> work. > >> >> There was no message in log. Geom reported no errors. Disk's has no > >> error. > >> >> It seems that there was some introduced inconsistency in UFS. > >> >> My thoughts was to stop/kill samba, remove samba's jail, unmount > >> partition > >> >> and do fsck. > >> >> > >> >> But 'jail -r' exit with something 'rc.shutdown exited ... 9' and le= ft > >> jail > >> >> running. pskill -KILL for samba processes say nothing and do nothin= g. > >> >> 'umount -f' say 'Device is busy'. Various utilities such as 'df' ha= ngs. > >> So > >> >> I forced to shutdown the server which has other important and worka= ble > >> >> service to resolve the situation. > >> >> > >> >> I wander is there any way to treat such a cases? May be 'umount -f'= can > >> >> have more power... > >> >> > >> > >> > So no errors from geom? Or from CAM? what's the underlying storage > >> > hardware? And what's the wchan / straceback for the processes in 'D' > >> state? > >> > And do you know the location of the file that's waiting for I/O? > >> > >> No errors. UFS was on gmirror and 'gmirror status' said 'OK'. > >> Unfortunately I was unable to do meaningful investigation under stress= . > >> There was a number of smbd process in a jail which ignore killing and > >> anything else. > >> The first inconsistency found by fsck later was a file with mtime > >> corresponding to a first claim of a problem from one user. Hour later = there > >> was more claims and finally samba share stop working. As I understand= UFS > >> inconsistency grew over that hour. Because there was multiple smbd > >> processes each corresponding to different user there it is no one file > >> maked problem. > >> > >> > >> > There's a number of things that might cause this, but they typically= are > >> > noisy about it. There's a couple of deadlock issues that might cause > >> this, > >> > but getting some more information is needed to understand what might= be > >> > done to mitigate or prevent the deadlocks. > >> > >> > P.S. Previous situation was when I do simple (as I think) experiment= s > >> with > >> >> ggated and have forced to reboot server when 'mount' hung. > >> >> > >> > >> > I assume ggate isn't involved when this happens now, right? > >> > >> That's right. I note this because it seems that stoping ggate server > >> while running ggate client is the simplest way to reproduce the proble= m. > >> > >> P.S. I was really hope that "umount -f" awake waiting processes for > >> killing. > >> > >> > >> > >> > >> >> -- > >> >> Best regards, > >> >> Anthony Pankov mailto:anthony.pankov@yahoo= .com > >> >> > >> >> > >> >> > >> > >> > >> -- > >> Best regards, > >> Anthony > >> > >> > > > -- > Best regards, > Anthony > >