System hang on shutdown when running freebsd-update

[Colin Perkins]

On 15 Oct 2014, at 18:23, Kevin Oberman <rkoberman at gmail.com> wrote:

> On Wed, Oct 15, 2014 at 2:40 AM, Colin Perkins <csp at csperkins.org> wrote:
> On 14 Oct 2014, at 18:09, Kevin Oberman <rkoberman at gmail.com> wrote:
> > I thought that this was just a fluke, but it has now happened three times,
> > so I guess it's now out of the "fluke" class.
> >
> > I have upgraded several times recently to each 10.1 BETA and RC. After the
> > first install pass t install the kernel and modules, the system shutdown
> > freezes at the very end. I see the buffers synced to the disks and get the
> > "All buffers synced" message. Then it just hangs. The disks are not marked
> > as clean and are fscked after a reset and boot.
> >
> > There is not much between the "All buffers synced" message and the call to
> > vfs_unmountall(), so I suspect it is hanging in that call. I admit that I
> > am pretty much lost whenever I look at the VFS code and I have not put a
> > lot of effort going further. Just hoping that someone familiar with it
> > might have an idea.
> >
> > I have tried several reboots and all run normally. The problem only seems
> > to appear when upgrading the OS. It happened repeatedly when I tried to
> > reboot before doing the second "install" pass of freebsd-update, but not
> > after, so the kernel and world are not in sync. I am baffled as to what
> > could be going on, but it means I need to be at the system (a baby server)
> > when I upgrade, but not every time I upgrade. I know it happened on the
> > 10.0-RELEAASE to 10.1-BETA1 and 10.1-RC1 to 10.1-RC2 upgrades.
> >
> > Has anyone else seen this?
> 
> I’m seeing the same behaviour, most recently when moving to 10.1-RC1 (haven’t gone to -RC2 yet). The system is:
> 
> FreeBSD 10.1-RC1 #0 r272463: Fri Oct  3 01:47:10 UTC 2014
> root at releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
> FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
> CPU: AMD Opteron(TM) Processor 6274                  (2200.05-MHz K8-class CPU)
> Origin = "AuthenticAMD"  Id = 0x600f12  Family = 0x15  Model = 0x1  Stepping = 2
> Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
> Features2=0x1e98220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX>
> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
> AMD Features2=0x1c9bfff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,XOP,SKINIT,WDT,LWP,FMA4,NodeId,Topology,PCXC,PNXC>
> TSC: P-state invariant, performance statistics
> real memory  = 549755813888 (524288 MB)
> avail memory = 534559084544 (509795 MB)
> Event timer "LAPIC" quality 400
> ACPI APIC Table: <041112 APIC1739>
> FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs
> FreeBSD/SMP: 4 package(s) x 16 core(s)
> …
> 
> I do have IPMI loaded, unlike the other reports.
> 
> --
> Colin Perkins
> https://csperkins.org/
> 
> Paul Koch replied privately with a pointer to a seemingly unrelated message he sent to stable last month. Take a look at the several paragraphs at the end starting with "On a side note". I'm suspicious that the generation of the large upgrade on /var during the "upgrade" pass is causing the delay. It fits pretty well and, in normal operation, my server would never see this issue at all.
> 
> https://docs.freebsd.org/cgi/getmsg.cgi?fetch=326083+0+/usr/local/www/db/text/2014/freebsd-stable/20140907.freebsd-stable
> 
> Aside from fsyncing the files, I suspect just running "upgrade" waiting for a long time before doing reboot might prevent it from happening. I is likely relevant that the single partition on the system is a 500GB SU+J UFS.

The update to -RC2 worked without problem on my system. I did, however, wait until vmstat showed the disks being idle after running freebsd-update (this only took a few minutes).

That said, when updating to -RC1 previously I left the box to sit for maybe an hour at the “All buffers synced” stage before giving up and resetting it, so if it was just waiting for the disks to sync, it was taking a very long time to do so. 

> I need to research a bit on how freebsd does things as well as possible interaction with the large SU+J partition. I was already uncomfortable about the SU+J but went with it due to the time it would otherwise take to fsck the 500GB disk.
> --
> R. Kevin Oberman, Network Engineer, Retired
> rkoberman at gmail.com



-- 
Colin Perkins
https://csperkins.org/