I have been hacked (WAS: Have I been hacked or is nmap wrong?)

[Kilian Hagemann]

On Wednesday 18 January 2006 16:25, Will Maier pondered:
> On Wed, Jan 18, 2006 at 03:56:32PM +0200, Kilian Hagemann wrote:
> > I have never even heard of "frox" before, but after some googling
> > it turns out that it's a GPL'ed transparent ftp proxy...
>
> Where's it pointing?

No idea, I only went as far as trying to login anonymously using a console 
based ftp client. How could I find out?

> > Also, I said smtp ports were open on the machines in question, I
> > just verified that I can send emails via BOTH these systems even
> > though no sendmail/exim/whatever was ever installed by me and
> > sendmail_enable="None" on both.
>
> What do you see when you connect to the SMTP ports? Are they really
> mail servers, or just rogue services running on 25?

They are really mail servers, at least smtp for outgoing mails (don't know 
about incoming though). I used kmail to configure them as standard outgoing 
smtp mail servers and successfully sent myself two emails, one via each 
server. Surely a default, out of the box, unconfigured and 
sendmail_enable="None" sendmail process wouldn't allow for something like 
that, never mind the fact that the firewall is supposed to block ANY access 
from the outside (output of ipfw show is attached)

> > My servers have been compromised, fantastic. And that with an
> > initial firewall'ed setup that left NO open ports (I verified that
> > a while ago with nmap). So much for my impression that FreeBSD was
> > secure.
>
> My condolences; what you describe, though, doesn't really suggest
> that /FreeBSD/ is insecure. In the vast majority of these situations
> (and yes, I have found myself in your shoes before), the operator
> (you or I) is to blame.

Alright, I guest that's a fair assumption. But that's what this thread is 
about: What (if anything) did I do wrong?

> > How could this have happened? ipfw buffer overflow? Some other
> > unknown vulnerability?
>
> Ockham's razor: the simplest is also the most likely solution.
> You're running Samba; is there any chance that that service or your
> configuration of it could have opened a hole? How many people have
> user accounts on that box? Do you allow
> ChallengeResponseAuthentication on SSH? Key only?

Well, I didn't worry about samba because it's firewalled to the outside(unless 
some Windows virus on one of the LAN machines exploited a samba hole, is that 
likely?). There is only one single normal user account with an uncommon name 
and an impossible password(16 characters randomly generated from ASCII 
charset). ChallengeResponseAuthentication is commented out in sshd which I 
guess means it uses the standard PAM authentication. It also allows 
password/interactive authentication in addition to public key, I always use 
the former. I do admit that I have set "PermitRootLogin yes" but my root 
password is 9 characters with numbers and non-alphanumeric characters, so 
hard to brute-force.

In any case, it's important to note that the only access from the outside via 
ssh/rsync is firewalled in such a way that it only allows access from a 
single IP address which my institution assigns me statically via DHCP (see 
attachment). They would have had to a) find out what this one and only 
trusted IP address is b) spoof it successfully c) attack ssh brute force?

> > I really wanna find out how they got in (syslog offers no clues
> > btw, I've been rootkitted after all :-(
>
> You'll need to do a more sophisticated forensic analysis, then, to
> figure out what happened. Some basic questions: were you running a
> file integrity monitor? What did it say? Do you have logs that were
> remotely backed up (and, therefore, likely still accurate)? What do
> they say? Do you have any network monitoring that might have
> recorded an intrusion? What services /should/ be running on the box
> (I don't think this was ever actually listed -- it would be useful
> to know)? Do you have dumps of the traffic leaving or entering the
> box?

Well, I thought my setup was secure enough for a very basic 
router/gateway/firewall for a couple of Windows machines using a sucky 
internet connection which is not worth stealing. So I didn't go through the 
effort of using a file integrity monitor, remote logging, traffic dumps or 
network monitors (jeez, sysadmins lives are really difficult these days :-( ) 
The services that should be running on the box are:

LAN only: samba, dnsmasq
LAN and WAN: ssh/rsync

I wanted to use rsync with ssh authentication/remote shell to sync my /etc 
and /usr/etc to my workstation and then comparing the "update" with a static 
copy to find out if anything had changed. But before I could do that, the one 
server mysteriously had its ssh/rsync disabled and I didn't take a healthy 
copy of /etc of the other one to begin with :-(

> Again, this is a tough and very unfortunate position to be in -- I
> sympathize. It may very well not be worth the time it takes to fully
> investigate the source of the compromise. Real forensic analysis is
> outside most of our job descriptions; I know that my skillset
> doesn't cover it well enough. An inept investigation can be much
> worse than no investigation at all: consider (if you can afford it)
> bringing in someone who can do a quick, good job of it.

> > Any suggestions other than format/reinstall/tripwire?
>
> I can't think of any better ideas. Certainly, I'd add updating the
> system to your list. Even if the Security Alerts don't seem to
> effect your set up, I find it's good practice to apply them in a
> reasonable amount of time. At the very least, it keeps me in touch
> with my boxes and lets me develop a routine in case an alert does
> effect me.

Thanks a ton for the help and advice, I'll see what I can do.

-- 
Kilian Hagemann
-------------- next part --------------
# output of ipfw show. I edited it to remove all count rules(merely used for traffic accounting),
# some unreach rules to prevent some LAN clients from accessing the internet altogether
# and substituted some ip blocks for privacy purposes. LAN_NET is the LAN subnet,
# MY_OUTSIDE_IP is the unique and only ip address that is allowed to login from the outside via ssh/rsync

00100    0       0 allow ip from any to any via lo0
00200    0       0 deny log ip from any to 127.0.0.0/8
00300    0       0 deny log ip from 127.0.0.0/8 to any
00500    0       0 deny log ip from any to any not verrevpath in via tun0
02400    0       0 check-state
02500 4083 2273839 allow tcp from { LAN_NET or me } to any setup keep-state
02600  305   25468 allow ip from any to any via vr0
02700   77    8094 allow udp from { LAN_NET or me } to any keep-state
02800    1     485 deny log udp from any to any
02900 2367  391644 allow tcp from MY_OUTSIDE_IP to me dst-port 22,873 via tun0 setup keep-sta
te
03000   61    5068 allow icmp from { LAN_NET or me } to any
03100   62    5208 allow icmp from any to { LAN_NET or dst-ip me } icmptypes 0,3,11
03200   47    4536 deny log ip from any to any
65535    0       0 allow ip from any to any