FreeBSD 8.1 Stable Unreasanoble Rebooting
Ian Smith
smithi at nimnet.asn.au
Fri Sep 24 03:58:29 UTC 2010
On Thu, 23 Sep 2010, jhell wrote:
> On 09/16/2010 15:33, Michael BlackHeart wrote:
> > 2010/9/16 Jeremy Chadwick <freebsd at jdc.parodius.com>:
> >> On Thu, Sep 16, 2010 at 08:37:29PM +0400, Michael BlackHeart wrote:
> >>> Today I've got a pretty strange event. It looks like a reboot but
> >>> unreasonable as far as I see. Before server's uptime was over month,
> >>> it's sometimes have to reboot for kernel updates or somethings like
> >>> that. I've digen all logs and didn't find a reason, so here they all.
> >>>
> >>> auth.log
> >>> Sep 16 13:59:58 diablo sshd[2284]: Received signal 15; terminating.
> >>> Sep 16 14:04:26 diablo sshd[2290]: Server listening on 0.0.0.0 port 22442.
> >>>
> >>> cron - nothing
> >>> debug.log - nothing
> >>> dmesg - nothing
> >>>
> >>> messages
> >>> Sep 16 13:44:55 diablo transmission-daemon[7965]: Couldn't create
> >>> socket: Protocol not supported (fdlimit.c:651)
> >>> Sep 16 13:45:31 diablo last message repeated 5 times
> >>> Sep 16 13:47:23 diablo last message repeated 13 times
> >>> Sep 16 13:57:40 diablo last message repeated 51 times
> >>> Sep 16 13:59:48 diablo last message repeated 12 times
Michael, were these above messages to be expected?
> >>> Sep 16 14:00:18 diablo named[1575]: stopping command channel on 127.0.0.1#953
> >>> Sep 16 14:00:18 diablo named[1575]: exiting
> >>> Sep 16 14:00:18 diablo syslogd: exiting on signal 15
> >>> Sep 16 14:02:31 diablo syslogd: kernel boot file is /boot/kernel/kernel
> >>> Sep 16 14:02:31 diablo kernel: Copyright (c) 1992-2010 The FreeBSD Project.
> >>> {...}
> >>
> >> This sure looks like a legitimate reboot to me (e.g. shutdown -r now);
> >> note how your system daemons (named, syslogd) are being shut down with
> >> SIGTERM. You can check with "last" (shutdown/reboot vs. crash).
> >>
> >> <paranoid>
> >> I would highly recommend taking this machine offline and reinstalling
> >> the OS, in addition to newfs'ing all existing filesystems (restore from
> >> last known good backup). buildworld/installworld and
> >> buildkernel/installkernel may not be enough depending on what the
> >> individual did. It's likely the machine could be compromised in some
> >> way, especially if there's any service on it which is public-facing,
> >> regardless of authentication mechanisms you've deployed in front of it.
> >> </paranoid>
[..]
> > That looks reasonable
> > last says:
> > reboot ~ th 16 sen 14:04
> > reboot ~ th 16 sen 14:03
> > shutdown ~ th 16 sen 13:59
> >
> > and it's pretty good syncs with logs but there's no anybody access to
> > physical console 'cos it's not even plugged in. That's for the first.
> > Next, I've got, I believe, pretty strong passwords, and also root
> > can't log in directly, but wheel user also is in operators so he also
> > can reboot or shutdown, but there's no any attempts or successful
> > logins. All potentialy dangerous services run under their own
> > unprerileged users, and so on. Crontabs also doesn't contain scripts,
> > I prefer periodic system, and there's no anyway anything that cause
> > reboot.
> > Thing that worries me it that there were multiple reboots and shutdown
> > that goes up by itself without anyone pressing a button. And in
> > messages log there's fsck segment that indicates to unnormal shutdown
> > or reboot. It looks like it started to shutting down but was in some
> > case interrupted and after powering up it few times reboots itself.
> > But commonly FreeBSD doesn't reboot by it's own will.
> > The same hardware worked over a half a year under 8.0 stables without
> > such a problem. I just would like to understand from where this
> > problem comes up.
> > This machine doesn't contain any critical info so I'll wait for a bit.
> > Also I'd like to notice that recently I've tuned hdd's spindown exept
> > system hdd by atacontrol port, powerd and CPU frequency lowering in
> > idle, maybe something of this could cause this problem? And where
> > could I check this out?
>
> You might just want to go through your /etc/rc.d/*, crontabs,
> /etc/periodic/* and /etc/rc* to check for any commands that call
> shutdown(8) or reboot(8).
>
> Not really malicious but a rogue user that was once a staffer can plant
> a shutdown/reboot command in any one of the above matching files and
> have it run by at(1). This really sounds like the case here.
>
> 1). Check the above files. egrep -nr "(shutdown|reboot)" on those.
> 2). Inspect the files for at(1) reboot(8) shutdown(8) or paths to
> unintelligible binaries that have been setuid=0(u+s) owner=root.
> 3). Keep in mind a rogue staffer may have well cp(1) shutdown(8) to
> another command or created a script containing shutdown(8) to another
> command that matches another system command or has replaced one.
>
> Thats just a few things to go on for now. Others may have more to add to it.
Might help to turn on process accounting to log who ran what, when? Add
accounting_enable="YES" to rc.conf, (touch /var/account/acct; accton) to
begin accounting before/without rebooting.
Then something like 'lastcomm -eE [-f /var/account/acct.N] | less' to
perouse the|[some] day's doings. To adjust how many days to keep etc:
grep account /etc/defaults/periodic.conf
cheers, Ian
More information about the freebsd-stable
mailing list