two NIC on 2 core system (scheduling problem)

[Alexander Motin]

Bartosz Giza wrote:
>> The CPU time you see there includes much more then just a card handling
>> itself. It also includes CPU time of the most parts of network stack
>> used to process received packet. So if you have NAT, big firewall,
>> netgraph or any other CPU-hungry actions done with packets incoming via
>> em0 you will see such results.
>> Even more interesting is that if bge0 or fxp0 cards will require much
>> CPU time to send packet, this time will also be accounted to em0 process.
>> :)
> 
> WOW that is weird. Yes i have quite big ipfw firewall and also couple of rules 
> from pf. So you are saying that whole overhead from firewall is counted to 
> this em taskq process. This is really strange for somebody who don't know 
> about this.
> So what in case if i would have two em nic's. How would then overhead from 
> firewalls be counted ? Splited to two taskq processes ?
> And another really weird thing is that if i have other card their processing 
> are counted to tasq process of different NIC. Why is this done in such a way ?

There is no dedicated processes in system to handle routing, firewall, 
netgraph, etc. Many processes would lead to multiple context switches, 
bigger latencies and reduced performance. Instead most parts of network 
stack implemented in direct call fashion. So NIC receive interrupt 
initiates packet handling by doing stack function call, that function 
calls another and so on until packet will be completely processed and 
transmitted or queued. There is not so many exception from this rule, 
for example, dummynet which schedules part of activity to the timer 
events. So sometimes you still can see some activity from the swi:net, 
dummynet or some other threads. It also tells not so much about who 
really consumed the CPU, but more about who initiated that CPU consumption.

In case of two em NICs em0 thread will mostly show load produced by 
em0->em1 traffic processing and em1 - load produced by em1->em0 traffic.

-- 
Alexander Motin