netisr software flowid
Robert Watson
rwatson at FreeBSD.org
Mon Sep 27 08:52:22 UTC 2010
On Mon, 27 Sep 2010, Artemiev Igor wrote:
> What is the status for software flowid calculation? I found the old netisr2
> patch[1] from Robert Watson and took from there code for setting flowid in
> tcp_input with some changes[2]. It work for me very well (8.1-stable) - now
> the server can handle not transit traffic without drops up to 118Kpps 60MB/s
> incoming and up to 107Kpps 50MB/s outgoing, netisr dispatch packets via
> three threads by round-robin:
Hi Artemiev:
I have a large outstanding patch set in Perforce that goes quite a long way
further, implementing the RSS model found in many network cards and aligning
OS hash tables for connection lookup with RSS. Where the RSS hash is made
available by the driver, the patches are also able to implement link-layer
dispatch. They largely eliminate the possibility of cache line contention in
the TCP/IP input path (as long as the driver also avoids cache line
contention) on multi-queue cards.
One reason I haven't merged the earlier patch is that many high-performance
10gbps (and even 1gbps) cards now support multiple input queues in hardware,
meaning that they have already done the work distribution by the time the
packets get to the OS. This makes the work distribution choice quite a bit
harder: has a packet already been adequately balanced, or is further
rebalancing required -- and of so, an equal distribution as selected in that
patch might not generate well-balanced CPU load.
Using just the RSS hash to distribute work, and single-queue input, I am able
to get doubled end-host TCP performance with highly concurrent connections at
10gbps, which is a useful result. I have high on my todo list to get the
patch you referenced into the mix as well and see how much the software
distrbiution hurts/helps...
Since you've done some measurement, what was the throughput on that system
without the patch applied, and how many cores?
Robert
>
> 12 root -44 - 0K 336K CPU2 2 18:43 56.15% {swi1: netisr 2}
> 12 root -44 - 0K 336K RUN 3 18:41 54.49% {swi1: netisr 3}
> 12 root -44 - 0K 336K CPU0 0 18:39 50.39% {swi1: netisr 0}
> 12 root -68 - 0K 336K WAIT 1 8:01 18.07% {irq256: bge0}
>
> So, what the reason to exclude this code from final version?
>
> [1] http://www.watson.org/~robert/freebsd/netperf/20090523-netisr2.diff
> [2] http://gate.kliksys.ru/~ai/software_flowid.diff
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
>
More information about the freebsd-current
mailing list