KVM exhaustion from routing table "leaks"

[Shaun Jurrens]

Hi all,

I've been fighting with a long term problem with a box that does a good deal
of packet pushing for a /17 and a little ipf work as well.  One nic does ipnat
for a /24. The box does some static routing between 2 cisco routers and the 
routes are added via the rc.conf mechanism for static routes, e.g.:

static_routes="bla0 bla1 ...."
route_bla0=" -net 193.xxx.3.0 -netmask 255.255.255.0 193.xxx.19x.1x"
route_bla1=" -net 193.xxx.4.0 -netmask 255.255.255.192 193.xxx.19x.1x"

The problem is in the continuous growth of cloned routes in the routing table.
I've managed to allocate enough kva to keep the box up for an extended amount 
of time but, eventually, it chews up every bit of kva that it can and 
allocating new routes fails and it has even taken the box down on occassion.
This happens on other boxes as well with lesser traffic.

The box is running 4.7-RELEASE-p7, with 5 fxp nic's, (4 in use)

...
Timecounter "i8254"  frequency 1193182 Hz
Timecounter "TSC"  frequency 863678217 Hz
CPU: Pentium III/Pentium III Xeon/Celeron (863.68-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x686  Stepping = 6
  Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CM
OV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 671023104 (655296K bytes)
avail memory = 648257536 (633064K bytes)
Preloaded elf kernel "kernel" at 0xc031f000.
Preloaded elf module "ipl.ko" at 0xc031f09c.
...

So, now a little information over the current state of things:

nol33n0x:/#> netstat -arn | wc -l
  696714

Number of static routes:
nol33n0x:/#> netstat -arn | grep S | wc -l
      34

Number of static routes with -cloning set:
nol33n0x:/#> netstat -arn | grep Sc | wc -l
      34

Number of cloned routes (box is still running, so number has grown):
nol33n0x:/#> netstat -arn | grep W | wc -l
  696830

Use of KVA by routing table:
nol33n0x:/#> vmstat -m | grep routetbl
	...
     routetbl1394589196107K 196107K262144K  1465571    0     0  16,32,64,128,256

and a little more:
 Memory Totals:  In Use    Free    Requests
               204934K   4103K    264929580

Observations: 

Number of routes with 'Use' == 0 on fxp0 (nic to "default" router):
nol33n0x:/#> netstat -arn | awk '/fxp0/ { print $5 }' | grep -e '^0$' | wc -l
  294790
Number of routes with 'Ref' == 0 on fxp0:
nol33n0x:/#> netstat -arn | awk '/fxp0/ { print $4 }' | grep -e '^0$' | wc -l
       3

Number of routes with 'Use' == 0 on fxp1 (small /24):
nol33n0x:/#> netstat -arn | awk '/fxp1/ { print $5 }' | grep -e '^0$' | wc -l
       1

Number of routes with 'Use' == 0 on fxp2 (most of the rest of our /17):
nol33n0x:/#> netstat -arn | awk '/fxp2/ { print $5 }' | grep -e '^0$' | wc -l         49
Number of routes with 'Ref' == 0 on fxp2:
nol33n0x:/#> netstat -arn | awk '/fxp2/ { print $4 }' | grep -e '^0$' | wc -l
       7

How icmp redirect is handled:
nol33n0x:/#> sysctl -a | grep redir
net.inet.ip.redirect: 1
net.inet.icmp.drop_redirect: 1
net.inet.icmp.log_redirect: 0

Sysctl's on routing:
net.inet.ip.rtexpire: 2
net.inet.ip.rtminexpire: 2
net.inet.ip.rtmaxcache: 512  (these seem to have no effect whatsoever...)


Specific questions:

1. Why do statically added routes assume -cloning?
2. Forgive my ignorance, but why is -cloning necessary for the default route?
3. Although I haven't done an exhaustive comparison of the content of the 
routing table, why don't cloned routes with Use==0 time out?
4. There was a security advisory about a possible DoS dealing with -cloning 
and KVA exhaustion on an earlier -release, was the fix part of the breakage?
5. Manual removal of routes with 'Use'==0 does not free up kernel memory, why?


I'm starting to think the next hack I'm going to have to try is running routed
or zebra to manipulate the routing table more actively, even though this would
seem to be sort of giving in to the problem, instead of fixing (assuming I'm
not just imagining all of this).

Perhaps I'm just ignorant of how routing is supposed to work, if so, I'll take
my cluebat like a man. I haven't dug through the code because I _know_ I'm 
ignorant there.  Guess I could use a little help. Comments and/or suggestions 
welcome.


-- 
Med vennlig hilsen/Sincerely,

Shaun D. Jurrens
Drift og Sikkerhetskonsulent
IKT-Avdeling
Oslo Skoleetaten
Tel:    +47 2208 7394
Mobil:  +47 9820 8826

gpg key fingerprint: 007A B6BD 8B1B BAB9 C583  2D19 3A7F 4A3E F83E 84AE
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20030513/c2c73aa9/attachment.bin