From nobody Fri Nov 28 15:09:29 2025 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dHxXV2sjJz6Hf27 for ; Fri, 28 Nov 2025 15:09:14 +0000 (UTC) (envelope-from anthony.pankov@yahoo.com) Received: from sonic308-18.consmr.mail.ir2.yahoo.com (sonic308-18.consmr.mail.ir2.yahoo.com [77.238.178.146]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4dHxXT025lz455Q for ; Fri, 28 Nov 2025 15:09:12 +0000 (UTC) (envelope-from anthony.pankov@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="rTP/K2Za"; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of anthony.pankov@yahoo.com designates 77.238.178.146 as permitted sender) smtp.mailfrom=anthony.pankov@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1764342549; bh=L79py+891KuyJdJhs4OWNKjZXv2a4paZJPEeELwnAF4=; h=Date:From:To:CC:Subject:In-Reply-To:References:From:Subject:Reply-To; b=rTP/K2ZaJuFjzB7d+vopDS3NBa/88YUji6t5miYxPOxfAI5AJgLIHIO8b7wGoJ5JLx7jEt9mkEkFQ3v/mAIDRo7PgiosFKZBlWYscg5OJUz3ukL6NHv6Gd+6B+9Dze0Q1sp1or3pdHpDdsmLZxM4HBWj6M0RhcOhO+0938QkMl/YXjOFZyPXY49BYs4+ASSYzy9NuImY/4tKK68syPKada2yQJF/GmqPgR578rnwbEfGwe7PN9T84yVdjAq2ksxKRkJTMLEvQqzW2/ZzKQX4o6OYHE9Uuq+574pFXwunHB/7eWAWafTny4KunEtfxY24CHcNgJ/ucrnt0p+4x7Lqpw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1764342549; bh=Osti7qbDvWF03FHOZjl2p4tlkNGP5KRFZy6wCbqeYXN=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=RbgIZJ/Ssfh285A9dmdETQrEhi/AiFG56ArlC1yCb8NXU8fWGMPI/2ax2Ql5o0TlAkwPOCwtQ8g5cB3cO+t062mF5CmkdrUNHU9cNAw0U93AhCB0Zt7h1OPdnObPbrJWjZPl3cFoX5IbrFctEb8vMzfk5REF7oJJCx4GuNYi/ECbTKGS0ZzkgJGMX+pTDlMbKWP1KoXjFrKT3zpqjBRJ5Y3R2KVuVvUVp3t45sUP0Mf/GzchJ05qc89SMctXfG5s5gf67UaP9VYYuLMLLefYI44MPrwnPQ3TnbRR7kzBTG+V831TQMHrLcWNs0rx8DqM/Na+JdRs+Ya0CKS6Y3UwOQ== X-YMail-OSG: ZTCjbkYVM1nq4eLJMHruwRwHaZRvl906gKBFmHizxSEC5Z2P1Mefh8ObCfAAZoG YqVfzeiZhx3u7lN6DOUvF1JScnYVHZ7dKvnG8GOBEeI_GLUD.6SxFvqYjtSPRvl5HSW2WUyjPDN8 jEb5.QG733mxbHpHX9wAEm0QbNvzigD36qdPlo0Dr.cxxtEQZDsrplSVjat93fWKaSNt45ofbnFL 0jMgHWC7WHDUIvffs343kPu_SyyJ4g5nDEy0cO1JBGoCim_wXfacsU65vt3OzEPD6aSebbFD5hWi isaqDrTSLum4BpiSBH4t5P1i7e8R070EcpbVk8eAjtTStIWIBXK3oJurMYFHulKRA66c9j1a0LxY kR2_JpAnZyYORLgKVJA4Ian.QF6yK31mpz7AclatQVPpGjuDQszEfJ3zQQAh3IYu9V1kpirlPzHV sFTta3y.3Tfg7mSRVuvdF3gGoJvkJFsCwZosqkKzvM6.ZQVA0l.DAxf0hAmtk9nL3Uw.E7ZqAa7_ xL5cpIaO0Ab9nLfykiuT11onxLqHoNVsS8eUlvo4OzeF5136k810zoUfrpnEE_BseJCPLnROPsB0 gCcnanv5okL7AsRbo46Z3vKbgCRHh3lziC3lJeNwDsRRW1vzT.nxSpYOOEQdDcfwtuSMsDPfalyY DKHVu4n8Eu.Jvq4O8208RuspiP9eZ5s.s2Q098N1I15.rjN_DeRGVJa1ufNzhXGLQ5NsiVT_LgvI tFq2TMxQUstaWuk244UzrXmZ_ea5S2cMfsPp9mdYxcHv_eajQlyIK4FNZsHNH.uDguhrrbI44Rlx mgUy4yGgHAg5OrHf2jD52Yoa_4B3miWu9VDgwJu4gPvF7he4pZK70UHi.wG0GjlqZz6oj6EzkWA0 Qqltas1JhNM4POAEYzhldQZepGi4XIalMaiUcu_vfKnYrWx3VYUsTf2HKN1OoiDVk0bh_rANaaIH .lhRDsWz.eEm4PwtLmp1itHXzGXvm5Z6eH6CKLjZoWHndZA1_I9rShj4CcZgRlotyEOCO4OMQwGt uXI2QDFGp.L7PxXJY5AjMrcJg1SvBcXk9gtymHG3ENa2KQ_s746RDz3n2aDUbnmwnKKF5xEH21aN ZOvDoNt31ifJXy_NvWYsZm0cPsFrOURZZpkoEtAYXOK9E.bI3OiKmATXsbEX7HzETwC4yRKeGaFH LAlzexh8XiNoAmTs4Jo.JEgD_nYXFnIUVFMsT1BSQFC.ZN3Np6WZwc1wbOBdCqUTzXhtA0BdNROU 0POEzw4HMnyK5hoZ9ZVmDc6HVexKV770qBocCZcha5Bi1qi3fgl.0UKzOKRrpUDPjURjPdSrfKPj 7vTd_I4ossCjwSe6bW1vHzS5n8lmQFyTcaKHLiuRixAKrcR.fuZ2DfFVtIYy5yOkcbpwTF4R9LJp 65H6WZvzI9ziHYjE_N7O.5dxwAVEhw.IhnNMbWJNqsKgRQY2Zg_gizi6sGUu86iIJpH7tCRLwEyy zwerfM8PeUHTcsa_9mFR6jDUwBUPCO.3eYgNDSwjPoFjVzs7weO8pVgUyEYy_He20hoOkh87gke4 CPulPrsaZNT6NJx6_yCf71lPQ8isv4Ka2qsd48rEfU0Dtr.je53PL94qLaB4ArHM6FpBZwQRT6U6 TxlmWamgR4MF2hl6iDDKAJqOqoSoyzSS5GZzctvydafMPuyoPjpB0lq_uONHdIRMCD_RJ6LsuDO5 1XjOI1Lr_rG9v8WAd_04OQ2ZTUT01BusJ11Dth8MNNw.HqDZi6vsmHPVnM94Dfn9QQlyBWngAYy1 HQiL0zMWUOqiKB.AubiNI..kY50.u0ZsPd8TuDZFuXCdgHTL6GMOTd.kWTf.33MKmiEGyJIEZ3_t xWyc44HJoagYFp_4CWx8V6e17c.dEivkw39ZqUEPms4v9TM2Kdgl9qeD7xmKx1i7Kml1.fO4NUf. b.Pr7QuIerFcWqrQ3k54h1dAFECTKBIQr_34tOuSVeudHW9u5YwOaL4dmmthy_UykNIOc4bFk5jh nSuJXoDbcM11BsQPYqmBljWoO9bmVCtTLaqdT7vGnc1CcoEmLTyGEWD46_vnl8zZfFFddPKZBJKS UnJ7x6aCbpQ.nL2wjZQs1.Erid7jBx_vHP4r.yOvSK4USNFO514VEaoII4vO0Rzp_NpNY.Abg2DM pWEcKsA9H1ZKWLxIsIK6qcz5AXmDlCBRhe.mT8qNBZ1r_GXCmDHi8.kdBUH_Q.wdBgU2pkORV4MV yRK8e1nufL7Dkh4912VfTyG7. X-Sonic-MF: X-Sonic-ID: 7b3a7f19-e3da-4e0b-b656-abed19e67e60 Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.ir2.yahoo.com with HTTP; Fri, 28 Nov 2025 15:09:09 +0000 Received: by hermes--production-ir2-76fb978d99-js4tk (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID a560d5f64d695f96dfe2282ec6b1293f; Fri, 28 Nov 2025 15:09:08 +0000 (UTC) Date: Fri, 28 Nov 2025 18:09:29 +0300 From: Anthony Pankov Message-ID: <101576524.20251128180929@yahoo.com> To: Rick Macklem CC: Warner Losh , freebsd-hackers@freebsd.org Subject: Re: any way to recover from I/O hang? In-Reply-To: References: <458400081.20251126171621.ref@yahoo.com> <458400081.20251126171621@yahoo.com> <908785764.20251127131317@yahoo.com> <1777092313.20251128124844@yahoo.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Mailer: WebService/1.1.24794 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo X-Spamd-Bar: -- X-Spamd-Result: default: False [-3.00 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; FREEMAIL_TO(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[77.238.178.146:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:34010, ipnet:77.238.176.0/22, country:GB]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; BLOCKLISTDE_FAIL(0.00)[77.238.178.146:server fail]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[77.238.178.146:from]; TAGGED_RCPT(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] X-Rspamd-Queue-Id: 4dHxXT025lz455Q =0D=0AFriday, November 28, 2025, 4:22:16 PM, you wrote: >> >> I wander do kernel care about infinite I/O waiting? Neither the problem = nor previous situation with ggate doesn't seem like a deadlock for me. >> If I remember correctly with ggate reviving ggate server do revive 'hang= ed' client which also was unresponsible to -KILL and 'umount -f'. >> >> Anyway I think it's very desirable that 'umount -f' will apply more fo= rce and not give up with "device busy" message. >> >> I partially understand that "more force" is a complex problem. May be of= something like replacing mountpoint vnode structures by deadfs structures. > This is not easy to do (as in unlikely that anyone will ever > implement it). The big issue is "if in progress file systems > are aborted, data is lost and the file system structure gets > messed up". Even if people are willing to live with data > loss, there might be new cases that fsck(8) must handle. I don't think this will introduce new cases for fsck because those situatio= ns ended in cold reboot. System shutdown hangs before any 'syncing' phase. = Aborting in progress filesystem programmatically or doing power off seems f= or me as the same cases for fsck. May be I'm wrong. > The exercise of implementing this could be a major project, > since all sorts of msleep()s must be changed to return and > then all the code that calls them must handle that special case > without problems. There is also be lots of waiters on locks > and buffer cache blocks that would need to be dealt with. > I did "umount -N" for the NFS client, which does what you > are asking for and it took quite a while to implement (and > I wouldn't be surprised if there is still bugs that make it > fail in rare cases). NFS has the advantage that it does > not deal with file system structures like inodes and > indirect blocks, but it still wasn't easy to get all the > "waiters on buffer cache buffers" to error out, etc. From=20my point of view some vnode operation on a mountpont (UFS partition = for samba share) didn't return and stuck somewhere in UFS module. Which mak= e user process (smbd) hang. Then more users process reach something where U= FS stuck. Meanwhile other UFS partitions worked with no problem. This give = me an idea that there is a possibility to release all structures and drop a= ll buffers related to problematic mountpoint keeping server alive. As for server operator "deadlock" in this case mean that there are processe= s which can't be killed because of stuck in failed mount and 'umount -f' th= at can't unmount because of processes used the mount. So I'm here to find a way to give a schedule to a stuck processes just for = killing or to force 'umount -f' release stuck processes and destroy the mou= nt. > As Warner said "If you collect info during the hang and > post that, someone might be able to diagnose the > problem". > As a starting point, the output of "ps axHl" when the > hang occurs should show what the various processes > are waiting on. Once you have the "wchan" for the > processes/threads that are stuck, you can grep around > in the sources to find out what it is sleeping on. > It might also be a resource exhaustion situation, such > as "out of vnodes". I know that, when kern.maxvnodes is > exceeded, the system doesn't hang, but it can get so slow > that it appears hung. (That one is recognized by threads > with a wchan that starts with "v", but I cannot remember > the exact wchan.) No, other services on a server was working well. Sorry for disturbing, this just make some grievance for me because a lot of= work was made to do thing 'right' and to separate services on a server. > procstat -kk can also be useful, to figure out what locks > various processes are waiting for. > rick >> >> >> > On Thu, Nov 27, 2025 at 3:13=E2=80=AFAM Anthony Pankov >> > wrote: >> >> >> >> >> Thursday, November 27, 2025, 7:58:21 AM, you wrote: >> >> >> >> > On Wed, Nov 26, 2025 at 7:16=E2=80=AFAM Anthony Pankov > >> > >> >> > wrote: >> >> >> >> >> >> >> >> Recently I'm again facing a situation where some UFS (?) error on = one >> >> >> partition forced me to reboot whole server and interrupt people's = work. >> >> >> As I understand the brief is: >> >> >> >> >> >> Reason: A process in state 'D' (uninterruptible wait/sleep) is blo= cked >> >> on >> >> >> an I/O operation (e.g., waiting for a network file share, a disk >> >> operation, >> >> >> or a device that is no longer responding) at the kernel level. The >> >> process >> >> >> is not awaken to handle signals until the condition it is waiting = for is >> >> >> resolved. >> >> >> Solution: You cannot kill a 'D' state process with any signal, inc= luding >> >> >> SIGKILL (kill -9). The condition must be resolved (e.g., fixing the >> >> network >> >> >> connection, waiting for the I/O to time out, or potentially unmoun= ting >> >> the >> >> >> filesystem with umount -f if it's a mounted network share). In som= e rare >> >> >> cases involving buggy kernel code or hardware failure, a reboot ma= y be >> >> the >> >> >> only option. >> >> >> >> >> >> In my case there was a jailed samba which share one ufs-formatted >> >> >> partition. The samba processes hangs one by one and sharing has st= opped >> >> it >> >> >> work. >> >> >> There was no message in log. Geom reported no errors. Disk's has no >> >> error. >> >> >> It seems that there was some introduced inconsistency in UFS. >> >> >> My thoughts was to stop/kill samba, remove samba's jail, unmount >> >> partition >> >> >> and do fsck. >> >> >> >> >> >> But 'jail -r' exit with something 'rc.shutdown exited ... 9' and l= eft >> >> jail >> >> >> running. pskill -KILL for samba processes say nothing and do nothi= ng. >> >> >> 'umount -f' say 'Device is busy'. Various utilities such as 'df' h= angs. >> >> So >> >> >> I forced to shutdown the server which has other important and work= able >> >> >> service to resolve the situation. >> >> >> >> >> >> I wander is there any way to treat such a cases? May be 'umount -f= ' can >> >> >> have more power... >> >> >> >> >> >> >> > So no errors from geom? Or from CAM? what's the underlying storage >> >> > hardware? And what's the wchan / straceback for the processes in 'D' >> >> state? >> >> > And do you know the location of the file that's waiting for I/O? >> >> >> >> No errors. UFS was on gmirror and 'gmirror status' said 'OK'. >> >> Unfortunately I was unable to do meaningful investigation under stres= s. >> >> There was a number of smbd process in a jail which ignore killing and >> >> anything else. >> >> The first inconsistency found by fsck later was a file with mtime >> >> corresponding to a first claim of a problem from one user. Hour later= there >> >> was more claims and finally samba share stop working. As I understan= d UFS >> >> inconsistency grew over that hour. Because there was multiple smbd >> >> processes each corresponding to different user there it is no one file >> >> maked problem. >> >> >> >> >> >> > There's a number of things that might cause this, but they typicall= y are >> >> > noisy about it. There's a couple of deadlock issues that might cause >> >> this, >> >> > but getting some more information is needed to understand what migh= t be >> >> > done to mitigate or prevent the deadlocks. >> >> >> >> > P.S. Previous situation was when I do simple (as I think) experimen= ts >> >> with >> >> >> ggated and have forced to reboot server when 'mount' hung. >> >> >> >> >> >> >> > I assume ggate isn't involved when this happens now, right? >> >> >> >> That's right. I note this because it seems that stoping ggate server >> >> while running ggate client is the simplest way to reproduce the probl= em. >> >> >> >> P.S. I was really hope that "umount -f" awake waiting processes for >> >> killing. >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Best regards, >> >> >> Anthony Pankov mailto:anthony.pankov@yaho= o.com >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> Best regards, >> >> Anthony >> >> >> >> >> >> >> -- >> Best regards, >> Anthony >> >> --=20 Best regards, Anthony