snmpd strangeness

[Jeremy Chadwick]

On Wed, Nov 19, 2008 at 10:57:50AM -0500, John Almberg wrote:
> I just noticed something odd and am looking for ideas...
>
> As you can see from the top snippet below, snmpd is getting hammered by 
> something. As a comparison, the load averages for this quad-core  box are 
> usually close to zero.
>
> I'm not even sure I'm using snmpd for anything... not even sure what it 
> is, precisely.
>
> I'm digging into docs at the moment, but any ideas much appreciated.

I'm greatly concerned by the fact that you have a process on your
machine taking up 103% CPU time (possible on a quad-core machine),
taking up 2621MBytes of memory (RSS), yet you have no idea what it is,
what SNMP is, or why said process is running on your machine.  :-)

You can truss the pid to find out what it's doing, but based on the
above I'm not sure the truss output will be of much use to you.

I would recommend finding out who/what started it by looking at the ppid
of the process (ps -alx | grep 45136, then look at the 3rd column which
is the ppid; then do ps -alx | grep {ppid}).  It's very possible the
ppid will be 1, which is init, which means in this case it was probably
started by a script in /usr/local/etc/rc.d.

I would then recommend using gcore on the snmpd pid, which will write
out a very large file (~2.6GB) to $PWD.  You can then examine that
later.

I would then recommend killing it off, then go on a quest to find out
why net-snmpd is on your machine -- and equally as odd, why it's
running.  For this to start, something has to be in /etc/rc.conf to
initialise it.

There's also the possibility that the process running isn't snmpd at
all, but rather a binary of a hacker who has gained access to your box,
especially given that you have no idea what it is.

> last pid: 38974;  load averages:  1.24,  1.40,  1.58
> 342 processes: 6 running, 336 sleeping
> CPU states: 13.7% user,  0.0% nice, 13.9% system,  0.3% interrupt, 72.1% 
> idle
> Mem: 5997M Active, 596M Inact, 420M Wired, 206M Cache, 214M Buf, 457M  
> Free
> Swap: 16G Total, 123M Used, 16G Free
>
>   PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU  
> COMMAND
> 45136 root        1 104    0  2636M  2621M CPU5   4 254.1H 103.91% snmpd
> 37368 www         1  20    0   193M 46232K lockf  6   0:05  3.91% httpd
> 38819 identry     1 -32    0  7688K  2648K CPU0   0   0:02  1.61% top

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |