compromised machines and entire network health

[Gary D. Margiotta]

On Thu, 13 Jul 2006, Arie Kachler wrote:

> Hello,
>
> In the past several years, we have had a few incidents of servers of 
> customers that are compromised and then flood our entire network and bring 
> down almost everything. The sql slammer worm for example.
>
> Is there a solution to this? I know all computers should be kept up to date 
> so this does not happen, but most times customers are not as attentive to 
> patches as we sysadmins are.
> Assuming that there will always be machines with security issues, is there a 
> way to prevent a compromised computer to bring down an entire network?
>
> Any suggestions will be greatly appreciated.
>
> Arie Kachler

Firewall each machine, or see if you can do rate limiting on the machines 
to minimize the amount of traffic each machine can pump out at any given 
time.  You can try to do it at the machine level, or you can look for 
smart hardware such as smart switches or IDS systems that will do it for 
you.

We limit each port on our switch to 10Mbit (which shouldn't be able to 
flood the entire network and take it down... maybe slow it up a slight 
bit, but nothing catastrophic), and we have alarms to trigger when 
bandwith exceeds a certain threshold for a certain length of time.  The 
port gets shut off if the alarm fires, and the customer is advised of 
their problem, and is required to fix it before we allow it back on the 
network.  No exceptions.

We enforce a policy for customers to patch their machines themselves, or 
they have us do it for them as a managed service.  The customer is 
responsible for any damages related to any hacks/worms/mistakes, and the 
machines are removed from the network until they are fixed, either by them 
or by us standing on a console.  If they don't upkeep their systems on 
their own, we do it for them and charge them for it.  If they refuse to 
pay, we shut off their machine, confiscate their hardware, and go after 
them for any other time and materials related to the problem.  Mostly it 
doesn't get that far, but you have to be prepared for it with a published 
policy outlining these types of things.

Most customers get the point after they see the initial bill for damages 
their machine caused, and they just have us manage their systems for them, 
it's easier (and cheaper) for them, and safer for us, plus they are not 
responsible for any more damages if a machine we manage has a problem.

-Gary