two NIC on 2 core system (scheduling problem)

[Bartosz Giza]

Tuesday 28 of October 2008 19:10:43 Alexander Motin napisał(a):
> Bartosz Giza wrote:
> >> The CPU time you see there includes much more then just a card
> >> handling itself. It also includes CPU time of the most parts of
> >> network stack used to process received packet. So if you have NAT, big
> >> firewall, netgraph or any other CPU-hungry actions done with packets
> >> incoming via em0 you will see such results.
> >> Even more interesting is that if bge0 or fxp0 cards will require much
> >> CPU time to send packet, this time will also be accounted to em0
> >> process.

I have checked this and you are right. When i turned off ipfw; taskq process 
started to use less cpu. But still what is strange why processing from 
other cards are counted in em0 taskq ? This is quite strange and in that 
way em0 taskq process is using more cpu on one of the cores. So what i 
think the best would be to have only em NICs because processing of the 
packets would be splitted to those taskq processes is that right ?


> > WOW that is weird. Yes i have quite big ipfw firewall and also couple
> > of rules from pf. So you are saying that whole overhead from firewall
> > is counted to this em taskq process. This is really strange for
> > somebody who don't know about this.
> > So what in case if i would have two em nic's. How would then overhead
> > from firewalls be counted ? Splited to two taskq processes ?
> > And another really weird thing is that if i have other card their
> > processing are counted to tasq process of different NIC. Why is this
> > done in such a way ?
>
> There is no dedicated processes in system to handle routing, firewall,
> netgraph, etc. Many processes would lead to multiple context switches,
> bigger latencies and reduced performance. Instead most parts of network
> stack implemented in direct call fashion. So NIC receive interrupt
> initiates packet handling by doing stack function call, that function
> calls another and so on until packet will be completely processed and
> transmitted or queued. There is not so many exception from this rule,
> for example, dummynet which schedules part of activity to the timer
> events. So sometimes you still can see some activity from the swi:net,
> dummynet or some other threads. It also tells not so much about who
> really consumed the CPU, but more about who initiated that CPU
> consumption.

Ok, good to know. But how is counted firewall overhead when i would have 
only bge cards. They don't use taskq so i assume i would see this as system 
usage correct ?

> In case of two em NICs em0 thread will mostly show load produced by
> em0->em1 traffic processing and em1 - load produced by em1->em0 traffic.