System hang on shutdown when running freebsd-update

[Walter Hop]

Another update. I narrowed down the issue due to /sbin/init being replaced. This might be the magic that freebsd-update does to make the crash happen. (Although there might be more situations that cause a hang.)

I can completely reliably trigger the hang on a default 10.1-RELEASE install on UFS2 in VMware Fusion with the following procedure:

# chflags noschg /sbin/init
# cp -Rp /sbin/init /sbin/init2
# rm -f /sbin/init
# mv /sbin/init2 /sbin/init
# chflags schg /sbin/init
# reboot
=> Hang after "All buffers synced."

This looks useful because we don’t have to do a full freebsd-update to get the hang now.
I’d be interested to see if others can reproduce, because for me in VMware it’s 100% of the times.

It doesn’t happen on 10.0 kernel, also not on 10.1 ZFS, and not on 10.1 UFS2 with softupdates disabled.

We updated 15 machines to 10.1 with a modified upgrade procedure (first disable softupdates, then upgrade to 10.1, then re-enable softupdates). Without softupdates there’s no lockup.

So: 10.1 + UFS2 + softupdates + replacing /sbin/init = hang+100% CPU on next reboot/root unmount/root remount as readonly.

I don’t know if we can research more, I wonder what would happen on CURRENT but I don’t have time to build right now...

Cheers,
WH

> On 29 Nov 2014, at 13:17, Walter Hop <freebsd at spam.lifeforms.nl> wrote:
> 
> I’m revisiting this issue, since unfortunately I still have it more often than not when upgrading to 10.1-RELEASE.
> 
> As Kevin Oberman suspected earlier in the thread, the issue seems to lie in unmounting. The same hang occurs when dropping to single user mode and trying to re-mount root as readonly.
> 
> I’ve also had another unmount issue after upgrading to 10.1-RELEASE:
> 
>    All buffers synced.
>    softdep_waitidle: Failed to flush worklist for 0xfffff800027b4330
>    unmount of / failed (BUSY)
> 
> I’ve created a PR with the information I have: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195458
> 
> With the EOL date of FreeBSD 10.0 on the horizon, it’s making me a little skittish.
> Any ideas of experiments I can do to get more info out of a problematic box, or other options to take?
> 
> Thanks!
> WH
> 
>> On 28 Oct 2014, at 23:09, Walter Hop <freebsd at spam.lifeforms.nl> wrote:
>> 
>> I noticed this same hang after upgrading from 10.0-RELEASE to 10.1-RC3 in a VM running under VMware Fusion, so the problem appears still present.
>> 
>> I could only make it happen in the single uptime just after the system was freebsd-updated from FreeBSD 10.0 to 10.1-RC3.
>> 
>> Here is a screenshot: http://lf.ms/wait-for-reboot.png
>> 
>> It did not make any progress after 2 hours of waiting. When restarting the VM, the disk was dirty.
>> 
>> Some interesting facts:
>> - Note "swapoff: /dev/da0p2: Cannot allocate memory" in the screenshot which might pose a clue. I haven’t seen this normally.
>> - FreeBSD does respond to ping while it is busy, so it is not a complete "freeze".
>> - The VM is at 100% CPU while this is going on.
> 
> -- 
> Walter Hop | PGP key: https://lifeforms.nl/pgp
> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"

-- 
Walter Hop | PGP key: https://lifeforms.nl/pgp