Interrupt performance

[Slawa Olhovchenkov]

On Sat, Jan 29, 2011 at 02:43:07PM +1100, Bruce Evans wrote:

> On Sat, 29 Jan 2011, Slawa Olhovchenkov wrote:
> 
> > On Sat, Jan 29, 2011 at 07:52:11AM +1100, Bruce Evans wrote:
> >>
> >> To see how much CPU is actually available, run something else and see how
> >> fast it runs.  A simple counting loops works well on UP systems.
> >
> > ===
> > #include <stdio.h>
> > #include <sys/time.h>
> >
> > int Dummy;
> >
> > int
> > main(int argc, char *argv[])
> > {
> > long int count,i,dt;
> > struct timeval st,et;
> >
> > count = atol(argv[1]);
> >
> > gettimeofday(&st, NULL);
> > for(i=count;i;i--) Dummy++;
> > gettimeofday(&et, NULL);
> > dt = (et.tv_sec-st.tv_sec)*1000000 + et.tv_usec-st.tv_usec;
> > printf("Elapsed %d us\n",dt);
> > }
> > ===
> >
> > This is ok?
> 
> It's better not to compete with the interrupt handler in the kernel by
> spinning making syscalls, but that will do for a start.

In this programm inner loop don't contain any syscall.
Better varian -- loop with syscalls?

> > ./loop 2000000000
> >
> > FreeBSD
> > 1 process: Elapsed 7554193 us
> > 2 process: Elapsed 14493692 us
> > netperf + 1 process: Elapsed 21403644 us
> 
> This shows about 35% user 65% network.
> 
> > Linux
> > 1 process: Elapsed 7524843 us
> > 2 process: Elapsed 14995866 us
> > netperf + 1 process: Elapsed 14107670 us
> 
> This shows about 53% user 47% network.
> 
> So FreeBSD has about 18% more network overhead (absolute: 65-47), or
> about 38% more network overhead (relative: (65-47)/47).  Not too
> surprising -- the context switches alone might cost that.

For only 14K vs 56K interrupt. 152% more network overhead per one interrupt.
And I see drammaticaly less number of context switches in linux stats
(by dstat).

> BTW, even -current vs my version of FreeBSD-5.2 has 10-20% more network
> overhead (relative) for tx, apparently due to bloat in the network
> stack.  This apparently has nothing to do with hardware.  The slowdown
> is much the same with bge (heavily modified in my version) and em
> (barely modified).  One thing that I modify in both drivers is increase
> the tx ifq length by a massive amount (from about 512 to about 20000).
> This must be bad for overhead because such a large queue cannot fit
> in the L2 cache.  A large amount and perhaps more than half of NIC
> overhead consists of waiting for cache misses.  The slowdown in -current
> might be caused by minor bloat crossing a threshold and thus causing
> just one more cache miss every packet or 2.

I read disccursion about support DCA in FreeBSD, sound nice.

> >> Normal profiling works poorly (I see you found my old mail about high
> >> resolution profiling).  Linux might be misreporting the overhead for
> >
> > I think next server will be support PMC.
> > Report from PMC still poorly?
> 
> I should be adequate, but I prefer my version of perfmon which can
> count cache misses precisely for every function.  But without patches,
> perfmon is even more broken than high resolution kernel profiling.

Can I use your version of perfmon? How? I don't have expirence with
any kernel profiling.

> >> ...
> >> generate too many interrupts and don't have much or any way to control
> >> this.  Linux will certainly be about to handle 56K int/S better than
> >> FreeBSD since it doesn't have heavyweight interrupt threads AFAIK.
> >> FreeBSD also has "fast" interrupts, which are much like normal interrupts
> >> used to be in FreeBSD.  I don't know if your NIC driver uses these.  I
> >
> > re0: [FILTER]
> >
> > I think this is answer ([FILTER]), but I don't understand this answer :).
> 
> [FILTER] means "fast".  re used to unconditionally use "fast" interrupts
> and a task queue, which IMO is a bad way to program an interrupt
> handler, but yongari@ recently overhauled re (again :-) so that it now
> doesn't use fast interrupts by default for the MSI/MSIX case .  (BTW,

Ineresting, but I don't think this help for this case -- old PCI
chip, old CPU, old RAM, old matherboard -- all old. I don't try to get
wirespeed gigabit performance from this old box, I try to compare
relative performance FreeBSD vs Linux (in last time I got many
feedback about poor network performance FreeBSD vs Linux).

> it still bogusly uses INTR_MPSAFE for the fast interrupt bus_setup_intr().)
> The overhaul probably also reduces interrupt overhead if it works on
> your hardware, just be reducing the interrupt frequency.  I don't
> understand what moderates the interrupt frequency in the MSI case.
> 
> >> I don't really know if this is low-end, but guess all RealTeks are :-).
> >
> > FreeBSD support interrupt moderation on this chip, and chip support
> > TOE :)
> 
> The support was poor according to yongari@'s long messages about
> improving it.  With working interrupt moderation, you just don't get
> an interrupt rate of even 14KHz, except transiently.  4KHz would be
> all I would be happy with on a 1-core 1.6GHz CPU.  Since re is so
> primitive, yongari@ only managed to limit the rate to 20KHz.  (Is this
> only for the non-MSI case, with the MSI case better even before?)
> Linux with its reduced interrupt latency can handled your observed
> 56KHz without losing by so much.  My 2-core Athlon (Turion) with its
> low-end bge 5705, whose brokenness consists mainly of:
>      completely broken interrupt moderation
>      can't handle full 1Gbps -- saturates at about 300Mbps
>      CPU and/or DMA resources used for 300Mbps are about the same as for
>        a bge 5701 at a full 1Mbps
> saturates at about 100 KHz bge interrupts.  This many interrupts takes
> about all of 1 CPU to handle the hardware part and 20% of another to
> generate packets.  The system saturates in much the same way under
> WinXP.  I only recently got a version of Linux to boot on this system
> and haven't tried network performance tests under Linux on it.
> 
> Bruce
> _______________________________________________
> freebsd-performance at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"