CPU affinity with ULE scheduler

[Archimedes Gaviola]

On Tue, Nov 11, 2008 at 6:33 AM, John Baldwin <jhb at freebsd.org> wrote:
> On Monday 10 November 2008 03:33:23 am Archimedes Gaviola wrote:
>> To Whom It May Concerned:
>>
>> Can someone explain or share about ULE scheduler (latest version 2 if
>> I'm not mistaken) dealing with CPU affinity? Is there any existing
>> benchmarks on this with FreeBSD? Because I am currently using 4BSD
>> scheduler and as what I have observed especially on processing high
>> network load traffic on multiple CPU cores, only one CPU were being
>> stressed with network interrupt while the rests are mostly in idle
>> state. This is an AMD-64 (4x) dual-core IBM system with GigE Broadcom
>> network interface cards (bce0 and bce1). Below is the snapshot of the
>> case.
>
> Interrupts are routed to a single CPU.  Since bce0 and bce1 are both on the
> same interrupt (irq 23), the CPU that interrupt is routed to is going to end
> up handling all the interrupts for bce0 and bce1.  This not something ULE or
> 4BSD have any control over.
>
> --
> John Baldwin
>

Hi John,

I'm sorry for the wrong snapshot. Here's the right one with my concern.

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   17 root        1 171   52     0K    16K CPU0   0  54:28 95.17% idle: cpu0
   15 root        1 171   52     0K    16K CPU2   2  55:55 93.65% idle: cpu2
   14 root        1 171   52     0K    16K CPU3   3  58:53 93.55% idle: cpu3
   13 root        1 171   52     0K    16K RUN    4  59:14 82.47% idle: cpu4
   12 root        1 171   52     0K    16K RUN    5  55:42 82.23% idle: cpu5
   16 root        1 171   52     0K    16K CPU1   1  58:13 77.78% idle: cpu1
   11 root        1 171   52     0K    16K CPU6   6  54:08 76.17% idle: cpu6
   36 root        1 -68 -187     0K    16K WAIT   7   8:50 65.53%
irq23: bce0 bce1
   10 root        1 171   52     0K    16K CPU7   7  48:19 29.79% idle: cpu7
   43 root        1 171   52     0K    16K pgzero 2   0:35  1.51% pagezero
 1372 root       10  20    0 16716K  5764K kserel 6  58:42  0.00% kmd
 4488 root        1  96    0 30676K  4236K select 2   1:51  0.00% sshd
   18 root        1 -32 -151     0K    16K WAIT   0   1:14  0.00% swi4: clock s
   20 root        1 -44 -163     0K    16K WAIT   0   0:30  0.00% swi1: net
  218 root        1  96    0  3852K  1376K select 0   0:23  0.00% syslogd
 2171 root        1  96    0 30676K  4224K select 6   0:19  0.00% sshd

Actually I was doing a network performance testing on this system with
FreeBSD-6.2 RELEASE using its default scheduler 4BSD and then I used a
tool to generate big amount of traffic around 600Mbps-700Mbps
traversing the FreeBSD system in bi-direction, meaning both network
interfaces are receiving traffic. What happened was, the CPU (cpu7)
that handles the (irq 23) on both interfaces consumed big amount of
CPU utilization around 65.53% in which it affects other running
applications and services like sshd and httpd. It's no longer
accessible when traffic is bombarded. With the current situation of my
FreeBSD system with only one CPU being stressed, I was thinking of
moving to FreeBSD-7.0 RELEASE with the ULE scheduler because I thought
my concern has something to do with the distributions of load on
multiple CPU cores handled by the scheduler especially at the network
level, processing network load. So, if it is more of interrupt
handling and not on the scheduler, is there a way we can optimize it?
Because if it still routed only to one CPU then for me it's still
inefficient. Who handles interrupt scheduling for bounding CPU in
order to prevent shared IRQ? Is there any improvements with
FreeBSD-7.0 with regards to interrupt handling?

Thanks,
Archimedes