FreeBSD boxes as a 'router'...

[Barney Cordoba]

Given that busses are faster and SMP is better now I'd say that your
tests need some refreshing.

--- On Wed, 11/21/12, Jean M Vandette <vandj at securenet.net> wrote:

> From: Jean M Vandette <vandj at securenet.net>
> Subject: RE: FreeBSD boxes as a 'router'...
> To: freebsd-isp at freebsd.org
> Date: Wednesday, November 21, 2012, 12:52 PM
> Greetings all.
> 
> We did allot of testing a few years ago with and without
> polling
> at that time what we had found it was not the nic's it was
> the bus
> that locked and caused a problem.  The best we were
> able to get was 80~90MB
> throughput 
> before the box would lock and reboot.  We also found a
> single CPU ran
> circles around
> a dual CPU for this purpose.  At that time we were
> running 3 BGP sessions so
> lots of memory
> was needed to hold the 240K routing table entries (at the
> time).  CPU load
> was never an issue.
> We were unsuccessful and moved to a Juniper routing
> platform, we just could
> not 
> get fast enough I/O to the bus.
> 
> I know allot of the hardware out there is much faster now a
> days and
> cheaper.  
> 
> The best advice I can give you is build a current
> state-of-the-art machine
> bus speed is the biggest issue, faster is better, I/O is
> critical, you can
> have the fastest bus
> out there but if all you have is a PCI slot for example your
> dead in the
> water, it's just not fast enough.
> You cannot have a bottle neck.  You need to check with
> the mfg for the bus
> speed of the nic.  
> Just because the nic is built in many times it is just to
> save a expansion
> slot, the speed
> to the bus is not as fast as you might expect. 
> 
> Once you get the fastest of I/O to a fast bus put a current
> release and  
> then hammer it with tests to tweak it.  Expect high
> interrupt rates if not
> using polling.
> Once your results are what you expect and stable put it into
> production.
> 
> You have had allot of good tweaks and info given to you over
> the last few
> days
> so I won't repeat them.  
> 
> Good luck
> 
> Jean M. Vandette
> SecureNet Information Services
> 
> -----Original Message-----
> From: owner-freebsd-isp at freebsd.org
> [mailto:owner-freebsd-isp at freebsd.org]
> On Behalf Of John Fretby
> Sent: Wednesday, November 21, 2012 11:41 AM
> To: Victor Balada Diaz
> Cc: freebsd-isp at freebsd.org
> Subject: Re: FreeBSD boxes as a 'router'...
> 
> On 21 November 2012 14:57, Victor Balada Diaz <victor at bsdes.net>
> wrote:
> 
> 
> > I think you forgot to CC the list. I'll add it so you
> can get more 
> > answers.
> >
> 
> I did forget, thanks for that! :)
> 
> 
> > em(4) and igb(4) are both drivers for Intel NICs. They
> just have 
> > different capabilities. The sysctl you're asking for
> controls behavior 
> > of adaptive interrupt moderation. It's a recommended
> tuning for end 
> > hosts more than routers. You can read more about
> interrupt moderation 
> > on this
> > document:
> >
> > http://www.intel.com/design/network/applnots/ap450.htm
> >
> > em(4) NICs don't have all the capabilities of igb(4)
> ones. Some em(4) 
> > NICs have interrupt moderation (eg: 82574L) but not all
> of them do. If 
> > your em(4) card does have interrupt moderation you can
> tune it with:
> >
> > hw.em.rx_int_delay
> > hw.em.rx_abs_int_delay
> > hw.em.tx_int_delay
> > hw.em.tx_abs_int_delay
> >
> > Exchanging latency to get more throughput.
> >
> > You can take a look at this document explaining
> capabilities of 
> > different
> > NICs:
> >
> >
> > http://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phy
> > s-brochure.pdf
> >
> > You should ask supermicro what's the exact model
> they'll put on your 
> > server and then decide if it's OK for you.
> 
> 
> They are apparently:
> 
> em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port
> 0xf020-0xf03f mem
> 0xdfa00000-0xdfa1ffff,0xdfa25000-0xdfa25fff irq 20 at device
> 25.0 on pci0
> em0: Using an MSI interrupt
> ...
> em0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAG
> IC,VLAN_HWTSO>
> 
> 
> > About the interrupt storm: We've had various interrupt
> storms that 
> > were caused by different problems. The most common was
> a software bug 
> > with interrupts.
> > After
> > reporting on the lists it was fixed and we didn't have
> problems again.
> >
> > If you have a problem with high interrupts because too
> many small 
> > packets (eg a DoS), getting a card with interrupt
> moderation should 
> > help a lot. Most probably your problem with interrupt
> storms was 
> > caused by something else like a shared interrupt with
> other device or 
> > software bug. Without more analysis it's impossible to
> really say.
> >
> 
> I have some details from when it happened - it doesn't look
> like it was a
> shared interrupt issue - it just literally looks like the
> host came up, with
> a stampeding hurd of "other" hosts hitting it for services
> that weren't yet
> running, and it folded :(
> 
> That's why I was wondering if there was a similar sysctl for
> the em driver
> - in order to raise the number of interrupts the system
> allows, before
> declaring it "a storm".
> 
> 
> >
> > Keep in mind that i'm not an expert on this area, so
> you might get 
> > better answers on frebsd-net@ :)
> >
> > Hope it helps.
> >
> 
> It has - half the problem is there are *so* many options,
> combinations - and
> no matter what you pick, if you look them up enough you'll
> find someone
> finding fault with them, or casting doubts on their
> performance.
> 
> Doesn't really help when all you want is something that has
> a good chance of
> "working" :)
> 
> -Jon
> _______________________________________________
> freebsd-isp at freebsd.org
> mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-isp
> To unsubscribe, send any mail to "freebsd-isp-unsubscribe at freebsd.org"
> 
> _______________________________________________
> freebsd-isp at freebsd.org
> mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-isp
> To unsubscribe, send any mail to "freebsd-isp-unsubscribe at freebsd.org"
>