FreeBSD 8.1 Stable Unreasanoble Rebooting

Ian Smith smithi at nimnet.asn.au
Fri Sep 24 03:58:29 UTC 2010


On Thu, 23 Sep 2010, jhell wrote:
 > On 09/16/2010 15:33, Michael BlackHeart wrote:
 > > 2010/9/16 Jeremy Chadwick <freebsd at jdc.parodius.com>:
 > >> On Thu, Sep 16, 2010 at 08:37:29PM +0400, Michael BlackHeart wrote:
 > >>> Today I've got a pretty strange event. It looks like a reboot but
 > >>> unreasonable as far as I see. Before server's uptime was over month,
 > >>> it's sometimes have to reboot for kernel updates or somethings like
 > >>> that. I've digen all logs and didn't find a reason, so here they all.
 > >>>
 > >>> auth.log
 > >>> Sep 16 13:59:58 diablo sshd[2284]: Received signal 15; terminating.
 > >>> Sep 16 14:04:26 diablo sshd[2290]: Server listening on 0.0.0.0 port 22442.
 > >>>
 > >>> cron - nothing
 > >>> debug.log - nothing
 > >>> dmesg - nothing
 > >>>
 > >>> messages
 > >>> Sep 16 13:44:55 diablo transmission-daemon[7965]: Couldn't create
 > >>> socket: Protocol not supported (fdlimit.c:651)
 > >>> Sep 16 13:45:31 diablo last message repeated 5 times
 > >>> Sep 16 13:47:23 diablo last message repeated 13 times
 > >>> Sep 16 13:57:40 diablo last message repeated 51 times
 > >>> Sep 16 13:59:48 diablo last message repeated 12 times

Michael, were these above messages to be expected?

 > >>> Sep 16 14:00:18 diablo named[1575]: stopping command channel on 127.0.0.1#953
 > >>> Sep 16 14:00:18 diablo named[1575]: exiting
 > >>> Sep 16 14:00:18 diablo syslogd: exiting on signal 15
 > >>> Sep 16 14:02:31 diablo syslogd: kernel boot file is /boot/kernel/kernel
 > >>> Sep 16 14:02:31 diablo kernel: Copyright (c) 1992-2010 The FreeBSD Project.
 > >>> {...}
 > >>
 > >> This sure looks like a legitimate reboot to me (e.g. shutdown -r now);
 > >> note how your system daemons (named, syslogd) are being shut down with
 > >> SIGTERM.  You can check with "last" (shutdown/reboot vs. crash).
 > >>
 > >> <paranoid>
 > >> I would highly recommend taking this machine offline and reinstalling
 > >> the OS, in addition to newfs'ing all existing filesystems (restore from
 > >> last known good backup).  buildworld/installworld and
 > >> buildkernel/installkernel may not be enough depending on what the
 > >> individual did.  It's likely the machine could be compromised in some
 > >> way, especially if there's any service on it which is public-facing,
 > >> regardless of authentication mechanisms you've deployed in front of it.
 > >> </paranoid>
[..]

 > > That looks reasonable
 > > last says:
 > > reboot           ~                         th 16 sen 14:04
 > > reboot           ~                         th 16 sen 14:03
 > > shutdown         ~                         th 16 sen 13:59
 > > 
 > > and it's pretty good syncs with logs but there's no anybody access to
 > > physical console 'cos it's not even plugged in. That's for the first.
 > > Next, I've got, I believe, pretty strong passwords, and also root
 > > can't log in directly, but wheel user also is in operators so he also
 > > can reboot or shutdown, but there's no any attempts or successful
 > > logins. All potentialy dangerous services run under their own
 > > unprerileged users, and so on. Crontabs also doesn't contain scripts,
 > > I prefer periodic system, and there's no anyway anything that cause
 > > reboot.
 > > Thing that worries me it that there were multiple reboots and shutdown
 > > that goes up by itself without anyone pressing a button. And in
 > > messages log there's fsck segment that indicates to unnormal shutdown
 > > or reboot. It looks like it started to shutting down but was in some
 > > case interrupted and after powering up it few times reboots itself.
 > > But commonly FreeBSD doesn't reboot by it's own will.
 > > The same hardware worked over a half a year under 8.0 stables without
 > > such a problem. I just would like to understand from where this
 > > problem comes up.
 > > This machine doesn't contain any critical info so I'll wait for a bit.
 > > Also I'd like to notice that recently I've tuned hdd's spindown exept
 > > system hdd by atacontrol port, powerd and CPU frequency lowering in
 > > idle, maybe something of this could cause this problem? And where
 > > could I check this out?
 > 
 > You might just want to go through your /etc/rc.d/*, crontabs,
 > /etc/periodic/* and /etc/rc* to check for any commands that call
 > shutdown(8) or reboot(8).
 > 
 > Not really malicious but a rogue user that was once a staffer can plant
 > a shutdown/reboot command in any one of the above matching files and
 > have it run by at(1). This really sounds like the case here.
 > 
 > 1). Check the above files. egrep -nr "(shutdown|reboot)" on those.
 > 2). Inspect the files for at(1) reboot(8) shutdown(8) or paths to
 > unintelligible binaries that have been setuid=0(u+s) owner=root.
 > 3). Keep in mind a rogue staffer may have well cp(1) shutdown(8) to
 > another command or created a script containing shutdown(8) to another
 > command that matches another system command or has replaced one.
 > 
 > Thats just a few things to go on for now. Others may have more to add to it.

Might help to turn on process accounting to log who ran what, when?  Add 
accounting_enable="YES" to rc.conf, (touch /var/account/acct; accton) to 
begin accounting before/without rebooting.

Then something like 'lastcomm -eE [-f /var/account/acct.N] | less' to 
perouse the|[some] day's doings.  To adjust how many days to keep etc: 
grep account /etc/defaults/periodic.conf

cheers, Ian


More information about the freebsd-stable mailing list