CPU affinity with ULE scheduler

Mon Nov 10 23:02:20 PST 2008

On Tue, Nov 11, 2008 at 12:32 PM, Archimedes Gaviola
<archimedes.gaviola at gmail.com> wrote:
> On Tue, Nov 11, 2008 at 6:33 AM, John Baldwin <jhb at freebsd.org> wrote:
>> On Monday 10 November 2008 03:33:23 am Archimedes Gaviola wrote:
>>> To Whom It May Concerned:
>>>
>>> Can someone explain or share about ULE scheduler (latest version 2 if
>>> I'm not mistaken) dealing with CPU affinity? Is there any existing
>>> benchmarks on this with FreeBSD? Because I am currently using 4BSD
>>> scheduler and as what I have observed especially on processing high
>>> network load traffic on multiple CPU cores, only one CPU were being
>>> stressed with network interrupt while the rests are mostly in idle
>>> state. This is an AMD-64 (4x) dual-core IBM system with GigE Broadcom
>>> network interface cards (bce0 and bce1). Below is the snapshot of the
>>> case.
>>
>> Interrupts are routed to a single CPU.  Since bce0 and bce1 are both on the
>> same interrupt (irq 23), the CPU that interrupt is routed to is going to end
>> up handling all the interrupts for bce0 and bce1.  This not something ULE or
>> 4BSD have any control over.
>>
>> --
>> John Baldwin
>>
>
> Hi John,
>
> I'm sorry for the wrong snapshot. Here's the right one with my concern.
>
>  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
>   17 root        1 171   52     0K    16K CPU0   0  54:28 95.17% idle: cpu0
>   15 root        1 171   52     0K    16K CPU2   2  55:55 93.65% idle: cpu2
>   14 root        1 171   52     0K    16K CPU3   3  58:53 93.55% idle: cpu3
>   13 root        1 171   52     0K    16K RUN    4  59:14 82.47% idle: cpu4
>   12 root        1 171   52     0K    16K RUN    5  55:42 82.23% idle: cpu5
>   16 root        1 171   52     0K    16K CPU1   1  58:13 77.78% idle: cpu1
>   11 root        1 171   52     0K    16K CPU6   6  54:08 76.17% idle: cpu6
>   36 root        1 -68 -187     0K    16K WAIT   7   8:50 65.53%
> irq23: bce0 bce1
>   10 root        1 171   52     0K    16K CPU7   7  48:19 29.79% idle: cpu7
>   43 root        1 171   52     0K    16K pgzero 2   0:35  1.51% pagezero
>  1372 root       10  20    0 16716K  5764K kserel 6  58:42  0.00% kmd
>  4488 root        1  96    0 30676K  4236K select 2   1:51  0.00% sshd
>   18 root        1 -32 -151     0K    16K WAIT   0   1:14  0.00% swi4: clock s
>   20 root        1 -44 -163     0K    16K WAIT   0   0:30  0.00% swi1: net
>  218 root        1  96    0  3852K  1376K select 0   0:23  0.00% syslogd
>  2171 root        1  96    0 30676K  4224K select 6   0:19  0.00% sshd
>
> Actually I was doing a network performance testing on this system with
> FreeBSD-6.2 RELEASE using its default scheduler 4BSD and then I used a
> tool to generate big amount of traffic around 600Mbps-700Mbps
> traversing the FreeBSD system in bi-direction, meaning both network
> interfaces are receiving traffic. What happened was, the CPU (cpu7)
> that handles the (irq 23) on both interfaces consumed big amount of
> CPU utilization around 65.53% in which it affects other running
> applications and services like sshd and httpd. It's no longer
> accessible when traffic is bombarded. With the current situation of my
> FreeBSD system with only one CPU being stressed, I was thinking of
> moving to FreeBSD-7.0 RELEASE with the ULE scheduler because I thought
> my concern has something to do with the distributions of load on
> multiple CPU cores handled by the scheduler especially at the network
> level, processing network load. So, if it is more of interrupt
> handling and not on the scheduler, is there a way we can optimize it?
> Because if it still routed only to one CPU then for me it's still
> inefficient. Who handles interrupt scheduling for bounding CPU in
> order to prevent shared IRQ? Is there any improvements with
> FreeBSD-7.0 with regards to interrupt handling?
>
> Thanks,
> Archimedes
>

Hi Ivan,

Archimedes Gaviola wrote:
> To Whom It May Concerned:
>=20
> Can someone explain or share about ULE scheduler (latest version 2 if
> I'm not mistaken) dealing with CPU affinity? Is there any existing
> benchmarks on this with FreeBSD? Because I am currently using 4BSD

Yes but not for network loads. See for example benchmarks in
http://people.freebsd.org/~kris/scaling/7.0%20and%20beyond.pdf

[Archimedes] Ah okay, so based on my understanding with ULE scheduler
in FreeBSD-7.0, it only scale well with userland applications
scheduling such as database and DNS?

> scheduler and as what I have observed especially on processing high
> network load traffic on multiple CPU cores, only one CPU were being
> stressed with network interrupt while the rests are mostly in idle
> state. This is an AMD-64 (4x) dual-core IBM system with GigE Broadcom
> network interface cards (bce0 and bce1). Below is the snapshot of the
> case.

This is unfortunately so and cannot be changed for now - you are not the
first with this particular performance problem.

[Archimedes] Meaning, you still have to improve the ULE scheduler
processing network load? I have read some papers and articles that
FreeBSD is implementing parallelized network stack, what is the status
of this development? Is processing high network load can address this?

BUT, looking at the data
in the snapshot you gave, it's not clear that there is a performance
problem in your case - bce is not nearly taking as much CPU time to be
bottlenecking. What exactly do you think is wrong in your case?

[Archimedes] Oh I'm sorry this is not the right one. Here below,

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   17 root        1 171   52     0K    16K CPU0   0  54:28 95.17% idle: cpu0
   15 root        1 171   52     0K    16K CPU2   2  55:55 93.65% idle: cpu2
   14 root        1 171   52     0K    16K CPU3   3  58:53 93.55% idle: cpu3
   13 root        1 171   52     0K    16K RUN    4  59:14 82.47% idle: cpu4
   12 root        1 171   52     0K    16K RUN    5  55:42 82.23% idle: cpu5
   16 root        1 171   52     0K    16K CPU1   1  58:13 77.78% idle: cpu1
   11 root        1 171   52     0K    16K CPU6   6  54:08 76.17% idle: cpu6
   36 root        1 -68 -187     0K    16K WAIT   7   8:50 65.53%
irq23: bce0 bce1
   10 root        1 171   52     0K    16K CPU7   7  48:19 29.79% idle: cpu7
   43 root        1 171   52     0K    16K pgzero 2   0:35  1.51% pagezero
 1372 root       10  20    0 16716K  5764K kserel 6  58:42  0.00% kmd
 4488 root        1  96    0 30676K  4236K select 2   1:51  0.00% sshd
   18 root        1 -32 -151     0K    16K WAIT   0   1:14  0.00% swi4: clock s
   20 root        1 -44 -163     0K    16K WAIT   0   0:30  0.00% swi1: net
  218 root        1  96    0  3852K  1376K select 0   0:23  0.00% syslogd
 2171 root        1  96    0 30676K  4224K select 6   0:19  0.00% sshd

I was doing network performance testing with a traffic generator tool
bombarding 600Mbps-700Mbps traversing my FreeBSD system in both
directions. As you can see cpu7 is bounded to irq23 shared on both
network interfaces bce0 and bce1. cpu7 takes up 65.53% CPU utilization
which affects some of the applications running on the system like sshd
and httpd. These services are no longer accessible when bombarding
that amount of traffic. Since there are still more idled CPUs, I'm
concern about CPU load distribution so that not only one CPU will be
stressed.

Thanks,
Archimedes