hw.igb.num_queues default

[Barney Cordoba]

--- On Thu, 6/20/13, Andre Oppermann  wrote:

> From: Andre Oppermann 
> Subject: Re: hw.igb.num_queues default
> To: "Eugene Grosbein" 
> Cc: "freebsd-net at freebsd.org" , "Eggert, Lars" , "Jack Vogel" 
> Date: Thursday, June 20, 2013, 10:29 AM
> On 20.06.2013 15:37, Eugene Grosbein
> wrote:
> > On 20.06.2013 17:34, Eggert, Lars wrote:
> >
> >> real memory  = 8589934592 (8192 MB)
> >> avail memory = 8239513600 (7857 MB)
> >
> >> By default, the igb driver seems to set up one
> queue per detected CPU. Googling around, people seemed to
> suggest that limiting the number of queues makes things work
> better. I can confirm that setting hw.igb.num_queues=2 seems
> to have fixed the issue. (Two was the first value I tried,
> maybe other values other than 0 would work, too.)
> >>
> >> In order to uphold POLA, should the igb driver
> maybe default to a conservative value for hw.igb.num_queues
> that may not deliver optimal performance, but at least works
> out of the box?
> >
> > Or, better, make nmbclusters auto-tuning smarter, if
> any.
> > I mean, use more nmbclusters for machines with large
> amounts of memory.
> 
> That has already been done in HEAD.
> 
> The other problem is the pre-filling of the large rings for
> all queues
> stranding large amounts of mbuf clusters.  OpenBSD
> starts with a small
> number of filled mbufs in the RX ring and then dynamically
> adjusts the
> number upwards if there is enough traffic to maintain deep
> buffers.  I
> don't know if it always quickly scales in practice though.

You're probably not running with 512MB these days, so pre-filling isn't much of an issue.
4 queues is only 8MB of ram with 1024 descriptors per queue, and 4MB with 512.

Think about the # of queues issue. In order to have acceptable latency, you need to do 6k-10k 
interrupts per second per queue. So with 4 queues you have to process 40K ints/second
and with 2 you only process 20k.  For a gig link 2 queues is much more efficient.

"Spreading" for the sake of spreading is more about Intel marketing than it is about
practical computing.

BC

BC