13.0-BETA3 frequent file system (UFS) hangs

[Jack L.]

I'm also having the same exact issues across several systems on
13-BETA3 and 14-CURRENT as of last week.

I recently upgraded everything to the latest 13-STABLE and 14-CURRENT
but uptime is only a day or so and not sure if it's still a problem.


On Mon, Feb 22, 2021 at 12:07 PM Florian Smeets via freebsd-stable
<freebsd-stable at freebsd.org> wrote:
>
> Hi,
>
> on Sunday I upgraded a lightly loaded web server (Nginx+PHP-FPM+MariaDB)
> from 12.2-RELEASE-p3 to 13.0-BETA3. The server has been hanging every
> few hours (between 2-12). Until now the only remedy I found was a reboot.
>
> When the issue appears I can still read from the file system, I was able
> to touch a (new) file, but not delete it.
>
> load: 0.49  cmd: rm 78200 [biowr] 7.31r 0.00u 0.28s 1% 2192k
> mi_switch+0xc1 _sleep+0x1cb bwait+0x6e bufwrite+0x206 ffs_update+0x2d0
> ffs_syncvnode+0x552 softdep_prelink+0x14b ufs_remove+0x85
> VOP_REMOVE_APV+0x27 kern_funlinkat+0x2d5 sys_unlink+0x28
> amd64_syscall+0x10c fast_syscall_common+0xf8
>
> The system is UFS only.
>
> /dev/da0s1a on / (ufs, local, noatime, journaled soft-updates, writes:
> sync 19483 async 3361, reads: sync 24501 async 4191, fsid 374666591e63cded)
>
> For some of the reboots I see the following in the log before rebooting:
>
> Feb 21 15:21:57 web01 kernel: Waiting (max 60 seconds) for system
> process `vnlru' to stop... done
> Feb 21 15:21:57 web01 kernel: Waiting (max 60 seconds) for system
> process `syncer' to stop...
> Feb 21 15:21:57 web01 kernel: Syncing disks, vnodes remaining... 20
> fsync: giving up on dirty (error = 35) 0xfffff80003815b70: type VCHR
> Feb 21 15:21:57 web01 kernel:     usecount 1, writecount 0, refcount
> 1074 seqc users 0 rdev 0xfffff80004c3f000
> Feb 21 15:21:57 web01 kernel:     hold count flags ()
> Feb 21 15:21:57 web01 kernel:     flags ()
> Feb 21 15:21:57 web01 kernel:     v_object 0xfffff800030d0c60 ref 0
> pages 45500 cleanbuf 1071 dirtybuf 1
> Feb 21 15:21:57 web01 kernel:     lock type mntfs: EXCL by thread
> 0xfffffe00c33fc000 (pid 24, syncer, tid 100097)
> Feb 21 15:21:57 web01 kernel: 9 5 0 0 done
>
> Is this related to the thread "FreeBSD 13.0-BETA2 and slow IO" ?
>
> I have a stable/13 kernel on there now, but that also hung once already.
>
> Do I need to disable SU/SUJ or put a main kernel on there to get it stable?
>
> Thanks,
> Florian