Tuning for router performance

[Joe Greco]

> RELENG_5 as of Apr 03 21:33
> 
> * P4 3.0GHz / 2GB DDR400 RAM
> * Supermicro P4SCi with 2 on-board Intel gige
> * Additional 2-port Intel gige on 64-bit PCI-X
> 
> With this many interfaces I have disabled unnecessary devices in the
> BIOS such as USB to reduce IRQ sharing.

Interesting.

I'm using 4.11-RELEASE on a P4 3.0G / 1GB DDR400, on a SuperMicro P4SC8
(which is basically the P4SCi with SCSI).  No additional 2-port card.
It's a fairly similar setup.

No problems with high traffic rates.  (For the purposes of this discussion,
all traffic except ssh to the router is traffic /thru/ the router).

            input        (Total)           output
   packets  errs      bytes    packets  errs      bytes colls
    107439     0   72162014     144813     0  142032833     0
    101704     0   68046409     135931     0  128359873     0
    104645     0   65768204     141990     0  129334101     0
    112468     0   75986846     150024     0  144615912     0
    106497     0   72030483     142558     0  141768086     0
     99765     0   70045449     133374     0  132466526     0
     96067     0   63798208     125844     0  125487982     0
    101886     0   70674519     134926     0  134179234     0
    103760     0   69877737     135941     0  137523002     0
    112911     0   79450478     154102     0  151303406     0
    116146     0   77471891     156882     0  152534206     0

The apparent high output rate is due to vlan devices, where the output is
attributed to both vlan* and em1 (dedicated to vlans).  The input and the
output are actually both approximately the input numbers.

We had noticed at one point that we were starting to see em1 report drops
which were in turn being reported as errs by the vlan devices.  We were 
pushing right near a gigabit of traffic at the time, so I had written it
off as just pushing the router to the edge.  But let's re-examine that.

Now, if I start pumping more PPS, I do indeed see errs:

            input        (Total)           output
   packets  errs      bytes    packets  errs      bytes colls drops
     97267     0   66675536     122086     0  131168641     0     0
    104247     0   75413955     130001     0  142561178     0     0
     95617     0   64612833     121289     0  127119179     0     0
    253471     0   77582829     397208 10000  139671386     0 10000
    289454     0   71304600     483657  9443  129348488     0  9443
    288084     0   71948183     486654  8725  135213379     0  8725
    287875     0   68385123     491482  7460  123001416     0  7460
    290068     0   72583882     487009 10140  135995424     0 10140
    283690     0   66749355     489184  5460  120351717     0  5460
    287656     0   71174008     494372  6546  135130910     0  6546
    294036     0   73220341     485374 12531  131770769     0 12531
    292414     0   74153414     489359 10575  139045664     0 10575
    293152     0   72187276     485446 11945  129875438     0 11945
    288516     0   73740228     484401  8261  138960063     0  8261
    286625     0   70340602     476896 11569  126852274     0 11569
    226024     0   70356111     374375  3985  135568639     0  3985
     97423     0   67559069     123148     0  131424813     0     0
     99617     0   70842045     126110     0  134722149     0     0
    109440     0   75528823     139232     0  148738405     0     0
     99994     0   69608728     128823     0  130991145     0     0

This is running a little "udpblast" program that sends min size packets
in addition to the production traffic load on the router.

I find this interesting, because the aggregate traffic through the router
is clearly not anywhere near a gigabit.  So it does appear that there is
some sort of inadvertent cap on PPS here.

Now, a few notes.

1) I notice that /your/ errors are showing as input errors.  Mine seem to
   show up as output errors.

2) We need to remember that the design of the P4SC{8,i} is a bit crappy,
   in that the onboard ports consist of one CSA port (no problem here)
   and one PCI port - which Supermicro wisely placed on the 32-bit, 33 MHz
   legacy PCI bus.  This could potentially limit the throughput on that
   port.

3) The driver tunable EM_TIDV looks quite promising as a first guess as to
   what to tune for /my/ issue. 

4) I think playing with EM_MAX_TXD and EM_MAX_RXD may be useful to both you
   and I as well.

5) Don't know about FreeBSD 5.  You're on your own there.  :-)

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.