Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Paul]

I can't reproduce the 580kpps maximum that I saw when I first compiled 
for some reason, I don't understand, the max I get even with ULE and 
preemption
is now about 530 and it dips to 480 a lot.. The first time I tried it it 
was at 580 and dipped to 520...what the?.. (kernel config attached at end)
* noticed that SOMETIMES the em0 taskq jumps around cpus and doesn't use 
100% of any one cpu
* noticed that the netstat packets per second rate varies explicitly 
with the CPU usage of em0 taskq
(top output with ULE/PREEMPTION compiled in):
 PID USERNAME PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   10 root     171 ki31     0K    16K RUN    3  64:12 94.09% idle: cpu3
   36 root     -68    -     0K    16K CPU1   1   5:43 89.75% em0 taskq
   13 root     171 ki31     0K    16K CPU0   0  63:21 87.30% idle: cpu0
   12 root     171 ki31     0K    16K RUN    1  62:44 66.75% idle: cpu1
   11 root     171 ki31     0K    16K CPU2   2  62:17 56.49% idle: cpu2
   39 root     -68    -     0K    16K -      0   0:54 10.64% em3 taskq

this is about 480-500kpps rate.........
now I wait a minute and

 PID USERNAME PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   10 root     171 ki31     0K    16K CPU3   3  64:56 100.00% idle: cpu3
   36 root     -68    -     0K    16K CPU2   2   6:21 94.14% em0 taskq
   13 root     171 ki31     0K    16K RUN    0  63:55 80.18% idle: cpu0
   11 root     171 ki31     0K    16K RUN    2  62:48 67.38% idle: cpu2
   12 root     171 ki31     0K    16K CPU1   1  63:04 58.40% idle: cpu1
   39 root     -68    -     0K    16K -      1   1:00 10.21% em3 taskq


530kpps rate.......


drops to 85%.. 480kpps rate
goes back up to 95% 530kpps

it keeps flopping like this...........

none of the CPUs are 100% use and none of the cpus add up , like the cpu 
time of em0 taskq is 94% so one of the cpus should be 6% idle but it's not.
This is with ULE/PREEMPTION.. I see different behavior without 
preemption and with 4bsd..
and I also see different behavior depending on the time of day lol :)
Figure that one out

I'll post back without preemption and with 4bsd in a min
then i'll move on to the 32 bit platform tests