NET.ISR and CPU utilization performance w/ HP DL 585 using FreeBSD 7.1 Beta2

[Won De Erick]

> ----- Original Message ----

> From: Ivan Voras <ivoras at freebsd.org>
> To: freebsd-hackers at freebsd.org
> Sent: Tuesday, November 18, 2008 5:21:04 AM
> Subject: Re: NET.ISR and CPU utilization performance w/ HP DL 585 using   FreeBSD 7.1 Beta2
> 
> Won De Erick wrote:
> 
> > I compiled the following em driver for Intel NIC Pro (82571) w/ FreeBSD 7.1 Beta 2 on HPDL 585 machine having 16CPUs.
> > 
> > http://people.yandex-team.ru/~wawa/
> > 
> > With net.isr.direct=1, I made some changes on kthreads(default=2) for em0 and em1's rx.
> > 
> > dev.em.0.rx_kthreads: 6
> > ....
> > dev.em.1.rx_kthreads: 6
> > 
> > With these settings, the result is:
> > 
> > CPU:  0.0% user,  0.0% nice, 57.2% system,  3.6% interrupt, 39.2% idle
> > Mem: 17M Active, 7228K Inact, 156M Wired, 76K Cache, 21M Buf, 31G Free
> > Swap: 4096M Total, 4096M Free
> > 
> >   PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
> >    52 root        1  43    -     0K    16K CPU12  c  41:38 100.00% em0_rx_kthread_1
> >    51 root        1  43    -     0K    16K CPU3   3  41:38 100.00% em0_rx_kthread_0
> >    54 root        1 -68    -     0K    16K CPU2   2  39:39 100.00% em1_txcleaner
> >  1283 root        1  43    -     0K    16K CPU1   1  38:55 100.00% em0_rx_kthread_3
> >  1282 root        1  43    -     0K    16K CPU10  a  38:55 100.00% em0_rx_kthread_2
> >  1344 root        1  43    -     0K    16K CPU9   9  25:51 100.00% em0_rx_kthread_5
> >  1343 root        1  43    -     0K    16K CPU4   4  25:51 100.00% em0_rx_kthread_4
> >    12 root        1 171 ki31     0K    16K CPU14  e  44:28 91.70% idle: cpu14
> 
> This is very interesting. Do you see real performance (network
> throughput) increase?
> 

There's a little improvement on the throughput, but packet errors occurred on both interfaces.

# netstat -I em1 -w 1 -d
            input          (em1)           output
   packets  errs      bytes    packets  errs      bytes colls drops
     32494   483   23083087      15681     0   23719154     0    82
     30547   330   23104447      16062     0   23077442     0    44

# netstat -I em0 -w 1 -d
            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls drops
     19889   640   24144754      21307     0    8719922     0     0
     18071  2436   25966238      21088     0    8766995     0     0

Is there any other thing that I can tweak to solve the problem?