Need urgent help regarding security

[Mark Jayson Alvarez]

Good day again!!
 
 This has something to do with my previous email about finding an IRC bouncer installed into one of our freebsd servers(4.9). Someone suggested here to run a rootkit finder... I installed an rkhunter and eventually found an ascii text file inside the /dev/ named "saux" and to my surprise, it contains all of our username and passwords we used to login to other servers from that machine. Afterwards, we didn't even run the same root kit finder into other machines and just looked for that file(saux) and walla!! all machines have one!! We immediately killed all remote administration daemons and allow only root console access. Now we have a lot of work to do. more than 10 servers have been compromised founded the same file("saux") containing our passwords. Critical servers such as dns, proxy, mail etc. Even two of our cisco routers are 80% possibly compromised as well..
 
 The question is: Now what?? I guess we will be spending 7 days of work starting from this day till we have a properly created policies, not just for user accounts... but I guess for everything, as in everything. And it wouldn't be only for a short period of time...I'm sure though. The bigger question is: Where should we start? Investigate how the cracker got into the system? Why? perhaps we should bring back the server first into their functional state because hundreds of thousands of people are relying to them??  Or should we tell our Director first, in case he might wonder why he is not receiving his emails on Monday morning or cannot telnet into the cisco router?
 
 Now we have  a couple of inputs, we just have to figure out which is the proper combination. Here they are:
 
 1. Use private key for ssh logins (should bring the private key always... and if it is stolen.....)
 2. Use kerberos for ssh logins? useful for cisco telnet authentication too. Should we replace the existing radius for the routers? Do we have enough time? can we afford to run a compromised server while setting up these servers?
 3. Constantly upgrade third party softwares (ssh, ssl, apache, bind) etc.. (too much work.. there are so many of them(postgres, proftp, mysql, php) must be member of various security mailing lists and discussions).
 4. Constant Os upgrade(or should we shift to OpenBSD like one of our boss recommended(need to familiarize first, it is a *nix no problem... but it is still OpenBSD :)Also, was it really the 4.8 that has been hacked or the old version of BIND running on it? Anyway, its 6.0 now, guess we really have to upgrade now.
 5. Use nmap versioning etc. constantly check for unknown services (must audit all of the services running on every machine)
 6. Always compile into a jail environment
 7. Create a standard firewall ruleset template, (if it is a web server... uncomment this etc.)
 8. use a livecd... (use for binary trojaning)
 9. remote sysloging (I thought "-ss" flag is recommended?)
 10. Implement kernel secure level chflags(undeletable, firewall unchangeable)
 11. Use ip forwarding so that public servers will never again face the Internet directly( does this require a supers strong machine that will act as firewall? or perhaps an appliance(brand new) can we acquire this right away?
 
 What else?? Do you have anymore idea? Right now I am about to reformat one of our proxy server and install 6.0 on it. Perhaps I should check the squid config throughly...
 
 
 Suggestions are welcome... very much welcome.  I just need to collate everything. 
 
 
 
 
 
 
 
 
 
 
 


		
---------------------------------
 Yahoo! FareChase - Search multiple travel sites in one click.