Problem on re0

[Sami Halabi]

Hi,
>It seems you have really complex network configuration(ipfw(4),
>lagg(4), dummynet(4), bridge(4) and ipsec(4) etc).  Finding out
>simplest network setup that shows the issue would be required here.
hmm.. i actually don't use lagg nor bridge nor ipsec, i compile them for
any future use that might be.
however i use IPFW, the MPD port limit connected users via pptp/l2tp with
ipfw rules via netgraph (i must admit i don't see them when I use ipfw
show) and probably dummynet.

Archetecture is simple: bge0 -- connected to border router, bge1 connected
to cables company where customers connect
we are routing 172.16.0.0/12 to bge1, default gateway is our peer in the
other side of bge0.
installed port as pptp/l2tp server is MPD.

please help me its really making us troubles...

Sami

On Mon, Dec 19, 2011 at 4:26 AM, YongHyeon PYUN <pyunyh at gmail.com> wrote:

> On Sat, Dec 17, 2011 at 06:49:48PM +0200, Sami Halabi wrote:
> > Hi,
> >
> > I've moved to a new server with bge card driver, similar configuration:
> > /etc/sysctl.conf
> > net.inet.flowtable.enable=0
> > net.inet.ip.fastforwarding=1
> > kern.ipc.somaxconn=8192
> > kern.ipc.shmmax=2147483648
> > kern.ipc.maxsockets=204800
> > kern.ipc.maxsockbuf=2097152
> > hw.intr_storm_threshold=9000
> > kern.maxfiles=256000
> > kern.maxfilesperproc=230400
> > net.inet.ip.dummynet.pipe_slot_limit=1000
> > #net.inet.ip.dummynet.io_fast=1
> > net.link.ether.ipfw=1
> > kern.ipc.nmbclusters=409600
> > net.graph.recvspace=40960
> > net.graph.maxdgram=40960
> >
> >  Kernel
> > -----------
> > device          lagg
> > options         IPFIREWALL
> > options         IPFIREWALL_FORWARD
> > options         IPFIREWALL_VERBOSE
> > options         DUMMYNET
> > options         HZ=1000
> > options         TCP_SIGNATURE
> > device          crypto          # core crypto support
> > device          cryptodev       # /dev/crypto for access to h/w
> > options         IPSEC
> > options         DEVICE_POLLING
> > device if_bridge
> >
> > /boot/loader.conf
> > net.graph.maxalloc=128000
> > net.graph.maxdata=128000
> > net.graph.threads=4
> >
> > and i have similar problem.. i check each minute for ping and i see
> > sometimes loss of 50% or 100% (ping -c 2 -t 4 otherside.ip.com),
> > here are the sysctl dev.bge.1.stats:
> > dev.bge.1.stats.FramesDroppedDueToFilters: 0
> > dev.bge.1.stats.DmaWriteQueueFull: 2291350
> > dev.bge.1.stats.DmaWriteHighPriQueueFull: 0
> > dev.bge.1.stats.NoMoreRxBDs: 0
> > dev.bge.1.stats.InputDiscards: 0
> > dev.bge.1.stats.InputErrors: 0
> > dev.bge.1.stats.RecvThresholdHit: 124120300
> > dev.bge.1.stats.DmaReadQueueFull: 13470948
> > dev.bge.1.stats.DmaReadHighPriQueueFull: 962
> > dev.bge.1.stats.SendDataCompQueueFull: 0
> > dev.bge.1.stats.RingSetSendProdIndex: 349289174
> > dev.bge.1.stats.RingStatusUpdate: 260304688
> > dev.bge.1.stats.Interrupts: 260304688
> > dev.bge.1.stats.AvoidedInterrupts: 0
> > dev.bge.1.stats.SendThresholdHit: 0
> > dev.bge.1.stats.rx.ifHCInOctets: 2165908673
> > dev.bge.1.stats.rx.Fragments: 0
> > dev.bge.1.stats.rx.UnicastPkts: 237503495
> > dev.bge.1.stats.rx.MulticastPkts: 0
> > dev.bge.1.stats.rx.FCSErrors: 0
> > dev.bge.1.stats.rx.AlignmentErrors: 0
> > dev.bge.1.stats.rx.xonPauseFramesReceived: 0
> > dev.bge.1.stats.rx.xoffPauseFramesReceived: 0
> > dev.bge.1.stats.rx.ControlFramesReceived: 0
> > dev.bge.1.stats.rx.xoffStateEntered: 0
> > dev.bge.1.stats.rx.FramesTooLong: 0
> > dev.bge.1.stats.rx.Jabbers: 0
> > dev.bge.1.stats.rx.UndersizePkts: 0
> > dev.bge.1.stats.rx.inRangeLengthError: 0
> > dev.bge.1.stats.rx.outRangeLengthError: 0
> > dev.bge.1.stats.tx.ifHCOutOctets: 1956421618
> > dev.bge.1.stats.tx.Collisions: 0
> > dev.bge.1.stats.tx.XonSent: 0
> > dev.bge.1.stats.tx.XoffSent: 0
> > dev.bge.1.stats.tx.flowControlDone: 0
> > dev.bge.1.stats.tx.InternalMacTransmitErrors: 0
> > dev.bge.1.stats.tx.SingleCollisionFrames: 0
> > dev.bge.1.stats.tx.MultipleCollisionFrames: 0
> > dev.bge.1.stats.tx.DeferredTransmissions: 0
> > dev.bge.1.stats.tx.ExcessiveCollisions: 0
> > dev.bge.1.stats.tx.LateCollisions: 0
> > dev.bge.1.stats.tx.UnicastPkts: 347260508
> > dev.bge.1.stats.tx.MulticastPkts: 0
> > dev.bge.1.stats.tx.BroadcastPkts: 30306
> > dev.bge.1.stats.tx.CarrierSenseErrors: 0
> > dev.bge.1.stats.tx.Discards: 0
> > dev.bge.1.stats.tx.Errors: 0
> >
> > this driver gives more statics...
> >
> > please help me, this causes us serious problems with customers.
> >
>
> Given that you see the same issue with bge(4) it looks like the
> root cause is not in ethernet driver.  I also see no evidence of
> dropped frames from bge(4) hardware MAC statistics.
> It seems you have really complex network configuration(ipfw(4),
> lagg(4), dummynet(4), bridge(4) and ipsec(4) etc).  Finding out
> simplest network setup that shows the issue would be required here.
>



-- 
Sami Halabi
Information Systems Engineer
NMS Projects Expert