[freebsd-questions] Re: Machine locking up

[Tuc at T-B-O-H.NET]

Hi,

	Might be on track there. The default "background_fsck" is on and
the first time I didn't see any of the fsck syslog messages. On the second
reboot, I see them all. Oddly, on the reboot I see the first one done in
12 minutes from the boot, yet the prior boot went over an hour before it
locked up.

	I did run into some oddness though. I found it was doing the final
fsck on a 1.1TB filesystem, so I wandered into '/mountpoint/.snap' and did
an "ls -lt". It hung. When I did a CTRL-T, it told me it was in a spinlock.
I ran top and found I had 1/2 a dozen processes also in spinlock. I was
patient this time (And had other multiple VTYs logged in) so I waited it
out. All of a sudden all the spinlocks stopped, the processes and ls
completed, but it still was in pass 5 at 95%. It sat there for a while
before finally sending the output to syslog and <KNOCK KNOCK> things are
doing fair.  I'm letting the system have it easy for a few days before
I put a load back onto it (4 SETI).

	The 2nd part to recompile kernel/world... I don't worry too much
doing that. Its the "install" part that gives me the willies. I've done
that before on a system that was "acting up" and on reboot it just 
"acted dead". I remember I had to hand copy some libraries over 
after remaking AGAIN and things started to pick up. But scary as heck.
The system finally died totally about a month later.

		Thanks, Tuc

> 
> I suggest you reboot in single user mode and fsck -fy the mount points you 
> use.
> If the problem persists - you might recompile the kernel and world.
> 
> Regards,
> Ivailo Tanusheff
> 
> 
> 
> 
> 
> "Tuc at T-B-O-H.NET" <ml at t-b-o-h.net> 
> Sent by: owner-freebsd-questions at freebsd.org
> 24.09.2008 10:56
> 
> To
> freebsd-questions at freebsd.org
> cc
> 
> Subject
> Machine locking up
> 
> 
> 
> 
> 
> 
> Hi,
> 
>                  We had a power failure at the datacenter we colocate at, 
> and all our
> servers went down. 
> 
>                  One is acting up ever since. It rebooted but wasn't 
> accessible. I
> tried to log in via our KVM and got the login/pass, but when it did the
> /etc/motd and rest it locked up. CTRL-C and CTRL-\ didn't do anything. 
> The remote KVM could C-A-D and the machine rebooted.
> 
>                  Once it rebooted it seemed ok, but then locked up again 
> while in use.
> The odd thing was I remembered to try CTRL-T, and got :
> 
> load: 0.00 cmd: login 1188 [sysctl lock] 0.00u 0.00s 0% 176k
> 
>                  on the session I was trying to log in on, and on the 
> session I was
> doing a tail, I saw :
> 
> load: 0.00  cmd: tail 2013 [proctree] 0.00u 0.00s 0% 0k
> 
>                  I could tab between VTYs, but logins wouldn't accept 
> input, and
> I couldn't C-A-D this time around. I could telnet to a special port on it
> for monitoring and get a reply, but SSH/sendmail didn't reply.
> 
>                  Its a FreeBSD 5.5 (I can't upgrade it, sorry)....
> 
>                  Any clues what to look for?
> 
>                                                  Thanks, Tuc